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MEMORANDUM  FOR  NAVAL  INSPECTOR  GENERAL 

SUBJECT:  Report  on  Navy  Transition  of  Advanced  Technology  Programs  to  Military 
Applications  (Report  No.  D-2003-053) 


We  are  providing  this  report  for  review  and  comment.  We  considered 
management  comments  on  a  draft  of  this  report  when  preparing  the  final  report. 

DoD  Directive  7650.3  requires  that  all  recommendation  be  resolved  promptly. 
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If  possible,  please  provide  management  comments  in  electronic  format  (Adobe 
Acrobat  file  only).  Copies  of  the  management  comments  must  contain  the  actual 
signature  of  the  authorizing  official.  We  cannot  accept  the  /  Signed  /  symbol  in  place  of 
the  actual  signature.  If  you  arrange  to  send  classified  comments  electronically,  they  must 
be  sent  over  the  SECRET  Internet  Protocol  Router  Network  (SIPRNET). 

We  appreciate  the  courtesies  extended  to  the  audit  staff.  Questions  should  be 
directed  to  Mr.  Bruce  A.  Burton  at  (703)  604-9071  (DSN  664-9071)  or  Mr.  Roger  H. 
Florence  at  (703)  604-9067  (DSN  664-9067).  See  Appendix  D  for  the  report  distribution. 
Audit  team  members  are  listed  inside  the  back  cover. 
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Report  No.  D-2003-053  February  4,  2003 

(Project  No.  D2001  AB-0105.001) 

Navy  Transition  of  Advanced  Technology  Programs  to 
Military  Applications 


Executive  Summary 


Who  Should  Read  This  Report  and  Why?  Science  and  technology  officials  in  the 
Office  of  the  Secretary  of  Defense  and  the  Department  of  the  Navy  should  read  this 
report  because  it  evaluates  the  Navy’s  current  process  for  enhancing  the  likelihood  that 
emerging  technology  would  reach  the  warfighter. 

Background.  Congress  and  DoD  officials  have  voiced  concern  that  technology  has  not 
quickly  transitioned  to  the  warfighter.  In  1998,  the  Vice  Chief  of  Naval  Operations 
instituted  the  Future  Naval  Capabilities  process  to  focus  the  science  and  technology 
investment  on  achieving  future  capabilities  for  Naval  forces.  The  Future  Naval 
Capabilities  process  was  designed  to  align  and  partner  the  requirements,  acquisition,  and 
science  and  technology  communities  to  focus  on  delivering  and  transitioning  priority 
Naval  capabilities.  To  help  achieve  the  objective,  the  Navy  funded  about  225  separate 
advanced  technology  projects  for  FY  2002  for  a  total  of  $870  million.  The  audit 
examined  39  science  and  technology  projects  (33  science  and  technology  products  and 
6  advanced  technology  demonstrations)  funded  with  research,  development,  test,  and 
evaluation  funds;  advanced  technology  development  appropriations. 

Results.  Although  the  Office  of  Naval  Research  created  a  structure  to  manage  its  science 
and  technology  efforts  to  facilitate  the  transition  of  technology,  improvements  are  still 
needed.  Specifically,  while  30  of  33  technologies  reviewed  did  have  working-level 
integrated  product  teams,  all  of  the  33  technologies  lacked  one  or  more  of  the  elements 
for  transitioning. 

•  None  of  the  technologies  had  integrated  product  teams  charters  to  establish  roles 
and  responsibilities, 

•  Not  all  recipients  were  included  in  the  working-level  integrated  product  teams, 
documentation  of  integrated  product  teams’  issues  and  actions  were  limited,  and 
agreements  on  technology  readiness  levels  and  exit  criteria  were  lacking,  and 

•  None  of  the  five  acquisition  recipients  had  identifiable  funding  for  technologies 
scheduled  to  transition  in  FYs  2002  and  2003. 

Similar  coordination  problems  were  identified  for  the  advanced  technology 
demonstration  projects  included  in  this  review. 

Unless  the  Office  of  Naval  Research  improves  its  coordination  with  plan  recipients  by 
establishing  working-level  integrated  product  teams,  and  evaluates  whether  near-tenn 


planned  transitioning  for  products  should  continue  if  transition  funds  are  lacking,  the 
Navy  cannot  make  fully  informed  and  prudent  decisions  on  whether  continued 
investment  is  warranted  (finding  A). 

The  Office  of  Naval  Research  did  not  use  the  performance  appraisal  process  effectively 
to  assist  in  achieving  DoD  performance  goals  and  its  corporate  goals  of  transitioning 
technology.  The  incorporation  of  technology  transition  in  performance  appraisal  plans  of 
product  managers  would  provide  accountability  and  contribute  to  the  likelihood  of 
technology  transitioning  (finding  B). 

The  Office  of  Naval  Research  did  not  have  an  effective  management  control  program  to 
evaluate  the  technology  transition  operations  within  the  Science  and  Technology 
Directorate.  Management  needs  to  establish  measurable  management  controls,  direct  the 
performance  of  internal  reviews,  and  provide  management  control  training  so  that  annual 
statements  of  assurance  are  based  on  reliable  assessments  of  the  science  and  technology 
process,  and  risks  and  controls  can  be  identified  (finding  C). 

Management  Comments  and  Audit  Response.  The  Deputy  Assistant  Secretary  of  the 
Navy  for  Planning,  Programming  and  Resources  provided  comments  to  the  draft  report 
and  concurred  with  most  of  the  recommendations.  The  Deputy  Assistant  Secretary 
nonconcurred  with  the  recommendation  to  review  the  advanced  technology  development 
efforts  for  five  unfunded  science  and  technology  projects  scheduled  for  near-term 
transition  and  discontinue  product  development  for  technologies  that  do  not  have  fonnal 
acquisition  program  support  including  identifiable  transition  funding.  See  the  Finding 
section  of  the  report  for  a  discussion  of  management  comments  and  the  Management 
Comments  section  for  the  complete  text  of  the  comments. 

Management  comments  were  generally  responsive  to  the  report  and  its  recommendations. 
However,  comments  in  response  to  the  recommendation  to  review  advanced  technology 
development  projects  that  lack  formal  acquisition  program  support  were  not  fully 
responsive.  Continued  expenditure  of  advanced  technology  development  funds  on 
technologies  that  do  not  have  coordinated  paths  or  plans  for  transitioning  to  acquisition 
programs  ignores  lessons  learned  and  training  on  successful  science  and  technology 
transitioning  provided  to  Department  officials.  Near-term  transitions  that  do  not  have  the 
necessary  commitment  by  the  scheduled  receiving  program  should  be  reviewed  to 
determine  whether  continued  development  is  warranted.  Although  the  Deputy  Assistant 
Secretary  concurred  with  the  recommendations  addressing  technology  transitioning  as  a 
performance  elements  in  the  appraisal  process  and  improving  the  management  control 
program,  the  proposed  actions  do  not  adequately  address  the  intent  of  the 
recommendations.  Accordingly,  we  request  additional  comments  from  the  Chief  of 
Naval  Research  on  the  final  report  by  March  6,  2003. 
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Background 


Defense  Acquisition.  The  Deputy  Secretary  of  Defense  issued  interim 
acquisition  guidance  on  October  30,  2002.  The  interim  acquisition  guidance 
states  that  science  and  technology  (S&T)  programs  shall  address  user  needs; 
maintain  a  broad-based  program  spanning  all  Defense-relevant  sciences  and 
technologies  to  anticipate  future  needs  and  those  not  being  pursued  by  civil  or 
commercial  communities;  preserve  long-range  research;  and  enable  rapid 
successful  transition  from  the  S&T  base  to  useful  military  products.  Advanced 
technology  shall  be  integrated  into  producible  systems  and  deployed  in  the 
shortest  time  practicable.  Teaming  among  warfighters,  users,  developers, 
acquirers,  technology  experts,  industry,  testers,  budgeting  officials,  and  system 
maintainers  shall  begin  during  requirements  definition. 

Science  and  Technology  Guidance.  An  affordability  task  force  chartered  by  the 
Director  for  Defense  Research  and  Engineering  issued  a  handbook  and  the 
Deputy  Under  Secretary  of  Defense  for  Science  and  Technology  issued  a  guide  to 
the  Military  Departments  and  Defense  agencies  concerning  practices  that  they 
believed,  if  instituted,  would  assist  in  transitioning  technology.  In  addition,  in 
response  to  congressional  concerns  that  the  DoD  had  not  been  successful  in 
transitioning  technology,  the  Under  Secretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics  issued  a  report  to  Congress  identifying  why 
technology  was  not  transitioning. 

Addressing  Affordability  in  Defense  Science  and  Technology  (S&T): 

A  Handbook  for  S&T  Managers.  In  October  1999,  the  DoD  S&T  Affordability 
Task  Force  endorsed  a  Handbook  that  stresses  the  importance  of  early 
involvement  of  all  candidate  acquisition  programs  in  advanced  technology  efforts. 
The  Handbook  states  that  early  involvement  of  advanced  technology  candidate 
acquisition  programs  in  research  development,  design,  test  planning, 
manufacturing,  training,  logistics,  financing,  and  contracting  are  essential  to 
address  key  issues  that  lock  in  a  majority  of  the  life-cycle  costs  of  programs.  The 
Handbook  states  that  management  tools  for  ensuring  effective  technology 
transitioning  include  establishing  integrated  product  teams  (known  as  IPTs), 
creating  IPT  charters,  identifying  quantitative  metrics  and  key  exit  criteria,  and 
developing  a  formal  transition  plan  that  is  officially  signed  by  the  “customer” 
(usually  an  acquisition  community  member)  and  the  technology  manager. 
Additional  management  tools  include  preparing  an  approved  memorandum  of 
agreement  or  understanding  that  includes  roles  and  responsibilities  of  the  various 
participants  and  a  funding  strategy,  which  commits  the  acquisition  community  to 
transition  the  technology. 

Technology  Transition  for  Affordability:  A  Guide  for  S&T  Program 
Managers.  In  April  2001,  the  Deputy  Under  Secretary  of  Defense  for  Science 
and  Technology  issued  a  Guide  to  provide  S&T  managers  with  strategies  to 
transition  technology  to  the  acquisition  community.  The  Guide  states  that  the 
transition  of  technology  should  be  timely  (get  the  technology  in  the  hands  of  the 
warfighter  as  soon  as  possible)  and  cost-effective  (provide  the  best  technology  at 
the  lowest  possible  cost).  The  Guide  states  that  a  key  strategy  for  transitioning 
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technology  is  early  coordination  between  the  S&T  project  manager  and  the 
receiving  acquisition  manager  to  promote  a  mutual  understanding  between  the 
two  parties. 

The  Guide  provides  that  IPTs  should  include  the  S&T  product  manager,  the  S&T 
contractor,  the  acquisition  manager  and  the  respective  contractor(s),  and  test  and 
evaluation  representatives.  An  IPT  should  be  formed  early  in  the  life  cycle  of  a 
technology’s  development  to  address  key  issues  that  can  greatly  affect  life-cycle 
cost  and  the  eventual  acceptance  and  implementation  of  the  technology.  Issues 
that  the  IPT  should  address  include  defining  and  agreeing  upon  quantifiable 
metrics,  such  as  cost,  performance,  and  schedule;  exit  criteria;  and  the  maturity  of 
the  technology  at  transition  identified  as  technology  readiness  levels  (TRLs)  (the 
TRLs  are  described  in  Appendix  B).  The  Guide  states  that  those  issues  and  others 
should  be  agreed  upon  in  formal  documentation  such  as  memorandums  of 
agreement  or  understanding  and  technology  transition  plans. 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 
Report  to  Congress.  In  June  2001,  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics  provided  a  Defense  Advanced  Research 
Projects  Agency  report  on  technology  transitioning  to  congressional  defense 
committees.  The  report  provided  Congress  with  the  results  of  a  review  of  the 
transition  of  research  to  the  Military  Departments  from  the  Defense  Advanced 
Research  Projects  Agency  and  addressed  issues  that  were  also  applicable  for  the 
Army  transition  of  research  technology  to  acquisition  program  managers  and, 
ultimately,  to  the  warfighter.  The  report  cited  a  key  reason  for  difficult 
technology  transition  was  the  need  for  collaboration  among  three  diverse  groups: 
the  S&T  researcher,  the  acquisition  program  manager,  and  the  military  user. 
Effective  transition  requires  the  groups  to  work  together  as  a  team,  which  is 
frequently  a  difficult  issue.  In  addition,  for  a  technology  transition  to  be 
successful,  the  acquisition  program  manager’s  prime  contractor  must  support  the 
technology  insertion,  and  the  technology  must  demonstrate  a  greater  return  than 
the  existing  capability. 

Future  Naval  Capabilities  Process.  In  1998,  the  Vice  Chief  of  Naval 
Operations  instituted  the  Future  Naval  Capabilities  (FNCs)  process  to  focus  the 
S&T  investment  on  the  achievement  of  future  capabilities  for  Naval  forces.  The 
Navy  adopted  the  process  in  November  1999.  The  FNC  process  was  designed  to 
align  and  partner  the  requirements,  acquisition,  and  S&T  communities  to  deliver 
and  transition  priority  Naval  capabilities  within  1  to  6  years.  At  the  center  of  the 
FNC  process,  an  IPT  for  each  FNC  brings  together  key  members  at  the  Flag 
Officer  or  Senior  Executive  Service  level  from  the  requirements,  acquisition,  and 
S&T  communities.  The  FNC  process  was  designed  to  bridge  the  gap  between  the 
acquisition  community  and  the  Office  of  Naval  Research  (ONR). 

The  Vice  Chief  of  Naval  Operations,  the  Assistant  Commandant  of  the  Marine 
Corps,  and  the  Assistant  Secretary  of  the  Navy  for  Research,  Development,  and 
Acquisition  approved  12  FNCs  that  concentrate  the  Navy’s  S&T  resources  on 
achieving  the  highest  priority  capabilities.  The  FNC  process  formally  began  in 
FY  2002. 
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Objectives 


The  audit  objective  was  to  determine  whether  the  Navy  was  successful  in 
transitioning  advanced  technology  projects  to  military  applications.  Specifically, 
we  detennined  whether  the  Navy  had  established  a  process  to  successfully 
transition  technology.  We  also  evaluated  management  controls  at  ONR  as  they 
relate  to  the  audit  objective.  See  Appendix  A  for  a  discussion  of  the  audit  scope 
and  methodology,  the  review  of  the  management  control  program,  and  prior 
coverage  related  to  the  audit  objectives. 

The  audit  examined  39  S&Ts  (33  S&T  products  and  6  ATDs)  funded  from  the 
advanced  technology  development  budget  activity  within  the  Navy  research, 
development,  test,  and  evaluation  funds;  advanced  technology  development 
appropriation.  The  39  S&T  efforts  had  expenditures  of  $152  million  in  FY  2002 
and  planned  funding  of  $430  million  through  FY  2007.  The  audit  examined 
whether  working-level  integrated  product  teams  were  established,  whether  the 
teams  included  the  planned  primary  and  secondary  recipients,  whether  charters 
were  established  for  the  teams,  whether  official  memorandums  of  understanding 
on  technology  development  were  established,  and  whether  agreements  included 
TRLs  and  exit  criteria. 
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A.  Navy  Science  and  Technology  Process 

While  executive-level  IPTs  were  established  for  the  12  FNCs, 
improvements  in  the  ONR  coordination  process  are  needed  at  the 
technology  working  level  and  at  the  acquisition  program  manager  level. 
Specifically,  although  30  of  33  technologies  did  have  working-level  IPTs, 
other  critical  elements  for  transitioning  were  missing: 

•  Working-level  IPTs  did  not  establish  charters  to  identify  roles  and 
responsibilities, 

•  Working-level  IPTs  did  not  include  all  planned  acquisition 
recipients, 

•  Documentation  of  IPT  issues  and  action  items  are  needed  to 
prevent  development  problems  resulting  from  key  personnel 
changes, 

•  Formal  agreement  on  TRLs  and  exit  criteria  were  not  established 
for  almost  half  of  the  technology  recipients,  and 

•  None  of  the  five  acquisition  recipients  identified  funding  for 
technologies  scheduled  to  transition  in  FYs  2002  and  2003. 

Similar  coordination  problems  were  identified  for  the  six  ATDs  also 
included  in  this  review.  These  conditions  exist  because  Navy  S&T 
management  did  not  require  formal  working-level  coordination  between 
acquisition  recipient  officials  and  Navy  S&T  officials  as  was  advocated  in 
the  Deputy  Under  Secretary  of  Defense  for  Science  and  Technology 
guidance.  In  addition,  the  ONR  had  not  established  minimum  standards 
for  technology  development  documentation.  As  a  result,  unless  measures 
are  undertaken  to  effectively  coordinate  the  39  technologies  that  have 
planned  additional  funding  of  $430  million,  the  Navy  cannot  make  fully 
informed  and  prudent  decisions  on  which  projects  warrant  continued 
investment. 

Navy  Advance  Technology  Program 

FNC  Planning  and  Execution.  To  maximize  the  benefit  to  the  warfighter,  ONR 
initiated  the  FNC  process  during  FY  1998.  The  ONR  action  required  shifting  the 
emphasis  from  conducting  S&T  research  on  very  long-term  development  of  high- 
risk  technologies  to  developing  focused  S&T  products  that  are  selected,  managed, 
and  demonstrated  to  meet  pre-established  exit  criteria  within  7  years.  Long-term, 
high-risk  research  continues  in  the  basic  and  applied  research  categories. 

The  Navy  Science  and  Technology  Corporate  Board  established  12  FNCs  that 
represent  the  highest  priority  performance  and  systems  to  be  delivered  to  the 
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warfighter  by  2010.  FNCs  were  to  be  managed  by  using  IPTs.  The  IPTs  for  the 
FNCs  function  like  a  corporate  board  and  are  composed  of  executive-level 
officials  from  the  Navy  or  Marine  Corps. 

The  goal  of  each  FNC  was  to  provide  significant  S&T  advances  in  products  and 
establish  deliverables  with  defined  demonstrations,  culminating  in  technology 
transitions.  S&T  products  are  the  fundamental  elements  of  the  FNCs.  S&T 
product  managers  were  to  establish  exit  criteria  to  support  transition.  Achieving 
the  exit  criteria  represents  development  success.  Exit  criteria  act  as  a  contract 
between  the  S&T  developer  and  the  customer. 

To  assist  in  achieving  the  goals  of  the  FNC  process,  ONR  established  the  require¬ 
ment  to  develop  business  plans  at  the  FNC  level.  The  business  plans  supported 
product  development  and  were  designed  as  a  management  tool  for  the  FNC  to 
maintain  program  control  at  the  product  level  and  document  the  expected 
performance  ranges  and  conditions  of  the  exit  criteria  for  the  emerging  S&T 
effort. 

Technology  Development  Guidance 


The  Deputy  Secretary  of  Defense  interim  guidance  states  that  S&T  programs  shall 
address  user  needs;  maintain  a  broad-based  program  spanning  all  Defense¬ 
relevant  sciences  and  technologies  to  anticipate  future  needs  and  those  not  being 
pursued  by  civil  or  commercial  communities;  preserve  long-range  research;  and 
enable  rapid,  successful  transition  from  the  S&T  base  to  useful  military  products. 
The  Handbook  provides  elements  of  best  practices  and  procedures  captured  in  the 
form  of  criteria  for  S&T  managers.  These  criteria  are  endorsed  by  the  Deputy 
Under  Secretary  of  Defense  for  Acquisition  and  Technology  Affordability  Task 
Force  that  was  charted  by  the  Director  for  Defense  Research  and  Engineering  to 
develop  solutions  on  how  to  strengthen  DoD  S&T  programs. 

Navy  Science  and  Technology  Products  Reviewed 


The  ONR  needs  to  strengthen  its  S&T  coordination  with  planned  acquisition 
program  recipients.  The  Handbook  and  Guide  recommend  the  establishment  of 
memorandums  of  agreement  or  understanding  with  the  acquisition  program 
manager(s),  including  agreements  on  TRLs  and  exit  criteria,  to  ensure  that 
acquisition  programs  have  the  necessary  funding  for  S&T  integration.  In  addition, 
ONR  should  require  S&T  product  managers  to  document  IPT  issues  and  action 
items  to  prevent  development  problems  resulting  from  key  personnel  changes. 

The  audit  examined  39  S&Ts  (33  S&T  products  and  6  ATDs)  funded  with 
research,  development,  test,  and  evaluation  funds;  advanced  technology 
development  appropriations  with  expenditures  of  $152  million  in  FY  2002;  and 
planned  funding  of  $430  million  through  for  FY  2007.  The  33  products  had 
35  primary  recipient  acquisition  programs  (1  product  had  two  additional  primary 
recipients)  and  the  6  ATDs  had  6  candidate  programs.  The  table  summarizes  the 
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audit  results  by  S&T  product  and  ATD,  using  the  recommended  business 
practices  in  the  Handbook  and  Guide. 


Summary  of  Science  and  Technology  Products  and 
Advanced  Technology  Demonstrations  (ATDs)  Examined 

(ratio  shows  positive  responses  to  total  examined)1 

Number  of  Occurrences  Percent  of  Occurrences 

33  Products2  6  ATDs3  33  Products  6  ATDs 


Integrated  Product  Team 
at  the  product  level 


Team  established 

30  of  33 

3  of  43 

91 

75 

Charter  established 

Oof  30 

Oof  3 

0 

0 

Acquisition  program 
manager  included 

30  of 354 

3  of  3 

86 

100 

Acquisition  program 
prime  contractor 
included 

21  of 255 

3  of  3 

84 

100 

Acquisition 
Program  Manager 


MOA/MOU/TTA 

23  of 354 

3  of  4 

66 

75 

Exit  TRLs  formally 
agreed 

12  of  35 

Oof  4 

34 

0 

Exit  criteria  formally 
agreed 

20  of  35 

2  of  4 

57 

50 

Funding  for  FYs  2002 
and  2003  transitions  by 
the  acquisition  recipient 

0  of  5 

0  of  0 

0 

0 

Appendix  C  provides  the  supporting  detail  to  the  table  by  FNC. 


1  Draft  documents  were  not  considered  as  a  positive  response. 

2  Reviewed  33  products:  14  of  Knowledge,  Superiority  and  Assurance;  6  of  Littoral 
Antisubmarine  Warfare;  5  of  Organic  Mine  Countermeasures;  and  8  of  Fleet  Force  Protection. 

3  We  reviewed  six  ATDs.  Of  the  six  ATDs,  four  were  scheduled  for  transitions  to  acquisition 
programs.  Two  were  scheduled  for  transition  to  an  FNC,  but  with  the  same  manager.  As  such, 
no  coordination  between  parties  is  necessary  for  these  two  efforts. 

4Thirty-three  products  identified  35  primary  acquisition  program  recipients  for  the  developing 
technologies  (1  product  in  Fleet  Force  Protection  had  2  additional  prime  recipients  identified). 
Thirty  of  the  33  recipients  were  included  in  the  working-level  IPTs. 

5Of  the  30  acquisition  program  recipients  included  in  working-level  IPTs,  25  had  contractors 
identified.  Twenty-one  of  the  25  contractors  were  participating  in  the  working-level  IPTs. 
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Integrated  Product  Teams 


The  Deputy  Secretary  of  Defense  issued  interim  acquisition  guidance  on 
October  30,  2002.  The  interim  acquisition  guidance  states  that  IPTs  will  be  used. 

IPTs  will  include  teaming  among  warfighters,  users,  developers,  acquirers, 
technology  experts,  industry,  testers,  budgeting  officials,  and  system  maintainers. 

Also,  the  Handbook  and  Guide  cite  best  business  practices  including 
establishment  of  working-level  IPTs  and  IPT  charters.  To  be  effective,  IPTs  must 
include  the  acquisition  program  manager(s)  and  have  an  established  charter.  The 
acquisition  program  manager(s)  prime  contractor  should  also  be  considered  for 
IPT  participation,  if  appropriate,  to  facilitate  the  technology  integration. 

Integrated  Product  Teams  Established.  The  Handbook  and  Guide  recommended 
the  establishment  of  IPTs  at  the  working  level  for  all  the  S&T  products  and  ATDs. 
ONR  officials  established  working-level  IPTs  for  30  of  the  33  products  and  for  all  the 
ATDs.  Thirty  of  the  33  primary  technology  recipients  were  participating  in  a 
working-level  IPT  to  some  degree.  In  addition  to  the  primary  recipients, 

1 1  technologies  had  secondary  recipients  that  represent  other  acquisition  programs 
identified  by  ONR  officials  that  also  are  candidate  systems  for  the  emerging 
technologies.  However,  ONR  documentation  showed  that  although  1 1  additional 
secondary  acquisition  programs  were  planned  to  receive  the  technology,  only 
1  participated  in  the  IPT.  Although  three  of  the  four  ATDs  had  established  a  working- 
level  IPT,  one  of  the  three  did  not  have  participation  by  the  primary  recipient.  For 
secondary  recipients  of  the  ATD  technologies,  one  of  the  three  were  participating  in 
the  working-level  IPTs. 

The  Handbook  and  Guide  do  not  distinguish  between  IPT  participation  at  the  primary 
and  secondary  level.  Although  the  focus  of  the  developing  technology  should  be  at  the 
primary  recipient  level,  secondary  recipient  participation  in  the  working-level  IPT  is 
important  to  preclude  avoidable  transition  cost  drivers  to  the  secondary  recipient.  It 
would  be  desirable  to  include  secondary  participants  in  technology  meetings  because 
both  primary  and  secondary  recipients  are  stakeholders  in  the  technology 
development,  and  primary  recipient  decisions  may  affect  the  secondary  recipient’s 
integration  of  the  technology. 

Integrated  Product  Team  Charters.  In  addition  to  the  establishment  of  IPTs, 
the  Handbook  and  Guide  recommend  the  establishment  of  IPT  charters.  The 
Handbook  states  that  IPT  charters  provide  the  best  way  to  minimize  team 
misunderstanding.  The  Handbook  provides  that  each  charter  should  include: 

•  The  mission  and  objectives  of  the  team, 

•  The  metrics  to  evaluate  the  team’s  progress, 

•  The  scope  of  the  team’s  responsibility, 

•  The  relationship  of  the  team  with  other  teams, 

•  The  authority  and  accountability  of  the  team, 
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•  The  resources  available  for  the  team,  and 

•  A  team  membership  list. 

Although  the  IPT  charters  to  identify  the  roles  and  responsibilities  of  the 
participants  have  been  established  at  the  executive  FNC  level,  none  of  the  S&T 
products  or  ATDs  had  established  IPT  charters. 

For  the  30  IPTs  established  for  the  33  products,  29  were  infonnal  IPTs.  The  IPTs 
usually  had  no  records  of  frequency,  attendance,  decisions,  or  outcomes  of 
meetings;  as  a  result,  the  audit  could  not  detennine  the  viability  of  the  IPTs. 
Specifically,  21  IPTs  did  not  document  discussions  or  decisions  resulting  from  the 
meetings. 

The  lack  of  documentation  for  IPT  meetings  may  provide  a  significant  risk  in  the 
orderly  continuation  of  S&T  efforts  because  25  percent  of  ONR  S&T  personnel 
will  be  eligible  to  retire  during  FYs  2002  through  2004.  In  addition,  ONR 
officials  stated  that  S&T  product  development  positions  were  also  filled  with 
detailees  from  field  offices  for  up  to  2  years.  The  combination  of  key  personnel 
changes  through  retirements,  detailees  returning  to  the  field  offices,  and  the  lack 
of  documentation  for  IPT  action  items  and  decisions  pose  a  development  risk 
because  of  the  potential  loss  of  corporate  knowledge.  The  lack  of  IPT 
documentation  occurred  because  ONR  did  not  provide  guidance  to  S&T  managers 
on  the  need  for  creating  records  of  IPT  meetings  and  results,  and  ONR 
management’s  belief  that  documentation  should  be  kept  to  a  minimum  to  reduce 
administrative  burdens.  This  risk  is  avoidable  by  requiring  the  documentation  of 
significant  program  decisions  and  development  issues.  ONR  does  not  require  the 
establishment  of  IPT  charters  to  document  the  roles  and  responsibilities  of  the 
ONR  or  acquisition  officials  or  require  the  establishment  of  a  record  of  issues, 
decisions,  or  action  items  resulting  from  the  IPTs. 

Acquisition  Program  Prime  Contractor.  One  goal  of  the  Navy  S&T  program  is 
to  transition  technology  to  an  acquisition  program  and,  subsequently,  to  the 
warfighter.  The  Guide  and  Handbook  recommend  that  the  acquisition  program 
prime  contractor  participate  in  the  IPT  to  facilitate  the  integration  of  the  evolving 
technology  into  the  receiving  platform.  The  30  products  that  established  IPTs  had 
33  primary  and  1 1  secondary  planned  technology  recipients.  The  33  primary 
recipients  had  25  prime  contractors  participating  in  their  acquisition  program. 
Prime  contractor  participation  in  the  working-level  IPTs  varied  significantly 
between  primary  and  secondary  recipients  in  that  21  of  the  25  primary  recipients’ 
contractors  participated  in  IPTs,  while  none  of  the  secondary  recipients’  prime 
contractors  participated. 

Of  the  three  ATDs  that  established  an  IPT,  all  primary  recipient  acquisition 
programs  had  prime  contractor  participation  in  the  IPT,  but  only  one  of  the 
three  secondary  recipients  had  their  prime  contractor  participating.  The  Navy  had 
no  official  S&T  guidance  that  discusses  representation  in  IPTs  for  products  or 
ATD  efforts. 
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Role  of  Acquisition  Program  Managers 


To  improve  the  technology  transition  rate,  acquisition  program  managers  must 
make  a  firm  commitment  to  transition  the  technology  to  their  programs.  The 
commitment  should  include  a  formal  and  up-to-date  memorandum  of  agreement 
or  understanding  between  the  S&T  product  manager  and  the  acquisition  program 
manager(s).  Each  memorandum  should  specify  the  relationships  and  the 
respective  responsibilities  of  the  S&T  product  manager  and  the  receiving 
acquisition  program  manager(s).  The  agreement  should  address  system 
requirements,  funding,  personnel  support,  exit  criteria,  and  TRLs.  Within  the 
ONR,  technology  transition  agreements  (TTAs)  represent  and  function  as  the 
memorandum  of  agreement  or  understanding.  The  TTAs  represent  agreement 
between  the  product  manager  and  the  planned  recipient  on  TRLs  and  exit  criteria. 
Although  ONR  guidance  requires  the  establishment  of  TTAs,  to  include 
agreement  on  TRLs  and  exit  criteria,  the  guidance  was  not  always  followed. 

Technology  Transition  Agreements.  The  ONR  issued  memorandum  guidance 
in  November  2001  that  required  the  development  of  TTAs  as  part  of  the  FNC 
business  and  execution  plans.  Each  advanced  technology  development  product 
was  to  have  a  TTA  that  represented  agreement  between  the  ONR  and  the 
receiving  acquisition  program  manager(s)  on  the  development  of  the  technology 
for  transitioning  and  the  exit  criteria.  For  the  33  products,  ONR  identified 
35  primary  and  1 1  secondary  recipients  for  the  emerging  product  technologies. 
Formal  TTAs  were  established  for  23  of  the  35  primary  acquisition  programs  and 
2  of  the  1 1  secondary  acquisition  programs.  The  four  ATDs  had  four  primary  and 
three  secondary  recipients  for  the  technologies  and  formal  agreements  were 
established  with  three  of  the  primary  and  two  of  the  secondary  recipients. 

Technology  Readiness  Levels.  DoD  adopted  TRLs  in  response  to  a  General 
Accounting  Office  Report,  “Best  Practices:  Better  Management  of  Technology 
Development  Can  Improve  Weapon  System  Outcomes,”  issued  in  July  1999. 
TRLs  are  an  assessment  of  the  technical  maturity  of  an  S&T  product  or  ATD.  In 
July  2001,  the  Deputy  Under  Secretary  of  Defense  for  Science  and  Technology 
issued  a  memorandum  to  the  Military  Departments  and  Defense  agencies  that 
emphasized  the  development  of  TRLs  for  S&T  efforts.  The  TRLs  range  from  one 
through  nine,  with  more  mature  S&T  efforts  having  a  higher  TRL  and  a  lower 
risk  for  the  acquisition  program.  The  ONR  had  not  established  TRLs  with  many 
of  the  planned  primary  or  secondary  technology  recipients.  Of  the  35  primary 
recipients,  only  12  agreed  to  the  TRLs,  and  no  agreements  were  established  with 
the  1 1  secondary  recipients. 

The  four  ATDs  had  four  primary  and  three  secondary  recipients  for  the 
technologies.  None  of  the  ATDs  had  agreements  on  the  TRLs  for  the  emerging 
technologies. 

Exit  Criteria.  The  exit  criteria  for  each  product  describe  the  current  capabilities, 
expected  performance  parameters  and  conditions  of  measurement,  the  range  of 
acceptable  perfonnance  improvements,  and  test  conditions  and  verification 
methods  for  measuring  performance.  The  S&T  product  manager  and  the 
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acquisition  program  managers,  in  collaboration  with  the  IPT,  develop  exit  criteria 
that  are  appropriate  for  transitioning  the  technology.  The  33  products  had 
35  primary  and  1 1  secondary  technology  recipients.  Agreement  on  the  exit 
criteria  was  established  for  20  of  the  35  primary  and  for  1  of  the  1 1  secondary 
recipients. 

The  four  ATDs  had  four  primary  and  three  secondary  recipients  for  the 
technologies.  Exit  criteria  agreements  was  made  with  two  of  the  primary  and  one 
of  the  secondary  recipients. 

Acquisition  Program  Funding 


The  DoD  research,  development,  test,  and  evaluation  budget  is  divided  into 
seven  budget  activities.  The  S&T  community  receives  funding  from  the  first 
three  budget  activities  only:  basic  research,  applied  research,  and  advanced 
technology  development.  The  acquisition  community  is  funded  with  three  budget 
activities:  demonstration  and  validation,  engineering  and  manufacturing 
development,  and  operational  systems  development.  A  seventh  budget  activity, 
management  support,  is  directed  toward  installations  or  operations  required  for 
general  research  and  development  use.  S&T  products  are  not  funded  from  the 
budget  activity  fund  appropriated  for  the  acquisition  community,  and  acquisition 
programs  are  not  funded  from  the  budget  activity  fund  appropriated  for  the  S&T 
community.  The  separation  of  research,  development,  test,  and  evaluation 
funding  between  the  S&T  and  acquisition  communities  and  the  shrinking  of  the 
research,  development,  testing,  and  evaluation  budget  make  coordination  between 
the  S&T  product  managers  and  acquisition  program  managers  very  critical.  If 
S&T  products  and  ATDs  are  critical  to  future  and  existing  weapon  systems,  and 
the  technology  is  successfully  demonstrated  using  coordinated  TRLs  and  exit 
criteria,  the  acquisition  community  must  set  funds  aside  for  transitioning. 

Acquisition  program  managers  were  not  providing  the  funding  for  technology 
transitioning.  For  the  33  products  reviewed,  5  products  were  scheduled  to 
transition  to  the  acquisition  program  manager  during  FY  2002  or  FY  2003.  Of 
the  5  products,  none  of  the  acquisition  program  managers  had  identifiable  funds 
for  the  transitioning  technologies.  Acquisition  program  managers  should  be 
required  to  specifically  identify  funds  for  the  emerging  technologies  particularly 
for  near-term  transitions  to  ensure  that  funds  are  available  and  to  determine  the 
adequacy  of  the  funding  as  a  requirement  for  continued  expenditure  of  advance 
technology  development  funds  by  ONR.  Without  adequate  funding  for 
technology  transitioning,  ONR  will  not  be  able  to  determine  whether  continued 
investment  in  S&T  products  is  beneficial  or  whether  the  limited  research  funds 
should  be  directed  to  other  more  promising  technology  transitioning  candidates. 
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Recommendations,  Management  Comments,  and  Audit 
Response 

A.  We  recommend  that  the  Chief  of  Naval  Research,  for  advanced 
technology  development  efforts: 

1.  Require  the  establishment  of  integrated  product  teams  that  may 
include  representatives  from  the  primary  and  secondary  candidate 
acquisition  program  office(s)  and  the  acquisition  program  office  prime 
contractor(s),  where  applicable. 

Management  Comments.  The  Deputy  Assistant  Secretary  of  the  Navy  for 
Planning,  Programming  and  Resources  partially  concurred  with  the 
recommendation.  The  Deputy  Assistant  Secretary  stated  that  the  Navy  is 
implementing  the  first  part  of  the  recommendation  to  include  target  acquisition 
program  offices  as  IPT  members.  The  Deputy  Assistant  Secretary  stated  that  all 
FNCs  have  IPTs  at  the  program  level  and  are  forming  IPTs  at  each  project  level. 
Further,  it  is  imprudent  to  require  IPTs  in  every  case,  but  exceptions  should  and 
will  be  rare,  justified,  and  documented. 

The  Deputy  Assistant  Secretary  stated  that  the  inclusion  of  prime  contractors  in 
all  IPTs  is  not  executable  and  potentially  harmful  to  the  acquisition  of  new 
technologies  because  the  IPT  is  a  forum  for  discussion  of  Government-only 
issues.  The  Deputy  Assistant  Secretary  stated  that  prime  contractors  are  included 
in  IPTs  when  the  program  manager  determines  it  is  appropriate.  The  Deputy 
Assistant  Secretary  suggested  that  the  recommendation  be  modified  to  include  the 
phrase  “with  the  acquisition  office’s  prime  contractor(s)  when  desired  by  the 
acquisition  program  manager.” 

Audit  Response.  The  comments  are  responsive  to  the  recommendation.  We 
agree  that  the  inclusion  of  the  acquisition  program  office  prime  contractor  in  all 
cases  may  not  be  appropriate  and,  as  such,  the  recommendation  included  the 
phrase  “where  applicable.”  As  a  technology  develops,  the  S&T  manager  and 
officials  receiving  the  technology  need  to  evaluate  whether  the  inclusion  of  the 
prime  contractor  is  warranted  when  technology  integration  considerations  need  to 
be  addressed.  Involvement  of  the  acquisition  prime  contractor  is  particularly 
important  when  the  acquisition  prime  contractor  is  different  than  the  ONR 
contractor  developing  the  emerging  technology  because  integration  technology 
issues  may  exist.  The  actions  discussed  in  the  Deputy  Assistant  Secretary’s 
comments  indicated  that  the  inclusion  of  the  acquisition  prime  contractor  will  be 
actively  considered  and  the  inclusion,  where  appropriate,  will  occur. 
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2.  Require  the  establishment  of  formal  memorandums  of  agreement 
or  understanding  and  technology  transition  agreements  between  the  science 
and  technology  manager  and  candidate  acquisition  program  manager(s). 

The  establishment  of  formal  agreements  should  be  a  requirement  for 
continued  research,  development,  test,  and  evaluation  funding. 

Management  Comments.  The  Deputy  Assistant  Secretary  concurred  and  stated 
that  all  FNCs  will  have  formal  memorandums  of  agreement  and  that  ONR  is 
implementing  a  requirement  for  memorandums  of  agreements  at  the  project  level. 
The  Deputy  Assistant  Secretary  stated  that  it  is  imprudent  to  require  a 
memorandum  of  agreement  in  every  ATD  project,  but  the  inability  to  reach  an 
agreement  with  an  acquisition  program  managers  becomes  an  indicator  of 
transition  problems  as  a  project  matures. 

3.  Review  the  technology  paths  or  plans  for  the  five  unfunded  science 
and  technology  products  that  are  scheduled  to  transition  in  FYs  2002  and 
2003  and  consider  discontinuing  product  development  that  does  not  have 
formal  acquisition  program  support  with  identified  funding. 

Management  Comments.  The  Deputy  Assistant  Secretary  nonconcured  and 
stated  that  the  intent  of  the  recommendation  goes  beyond  the  scheduled  transition 
in  FYs  2002  and  2003  in  that  the  principle  should  be  applied  in  future  cases.  The 
Deputy  Assistant  Secretary  stated  that  the  Navy  agrees  with  concerns  that  the 
Director,  Defense  Research  and  Engineering  expressed  to  the  Deputy  Under 
Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  in  response  to  a 
similar  recommendation  on  Army  research  programs  (Inspector  General,  DoD, 
Report  No.  D-2002-107,  “Army  Transition  of  Advanced  Technology  Programs  to 
Military  Applications,”  June  14,  2002).  The  Deputy  Assistant  Secretary’s 
comments  referred  to  statements  by  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics  that  to  bring  transformational 
technologies  to  operating  forces  depends  on  an  acquisition  environment  that 
fosters  efficiency,  creativity,  and  innovation  to  demonstrate  unproven 
technologies  and  explore  risk  mitigation  before  an  acquisition  program  commits 
to  a  technology.  The  Deputy  Assistant  Secretary  added  that  the  Chief  of  Naval 
Research  has  no  authority  to  compel  program  managers  to  identify  transition 
funding  and  that  all  ATD  programs  that  had  not  already  satisfied  the  criteria  for 
acquisition  would  have  to  be  discontinued  to  satisfy  the  recommendation.  Such 
an  action  would  severely  hamper  ONR  efforts  to  provide  innovative  solutions  and 
transformational  naval  capabilities. 

Audit  Response.  Management  comments  were  not  fully  responsive.  We  revised 
the  recommendation  based  on  the  comments.  The  report  of  the  Under  Secretary 
of  Defense  for  Acquisition,  Technology,  and  Logistics  to  Congress,  in  June  2001, 
cited  that  a  key  reason  for  difficulty  in  technology  transitions  was  the  need  for 
collaboration  among  three  diverse  groups:  the  S&T  researcher,  the  acquisition 
program  manager,  and  the  military  user.  The  Under  Secretary’s  report  stated  that 
effective  transitions  require  the  groups  to  work  together  as  a  team,  which  is 
frequently  difficult  because  of  their  diversity.  The  intent  of  the  recommendation 
was  to  address  the  five  near-tenn  technologies  in  question.  The  audit  results  for 
the  five  near-term  technologies  identify  that  coordination  needed  to  be  improved 
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and  that  funds  necessary  for  technology  transition  were  not  available.  We  agree 
with  the  Under  Secretary’s  comments  that  an  acquisition  environment  that  fosters 
efficiency,  creativity,  and  innovation  is  necessary  before  an  acquisition  program 
commits  to  a  technology.  However,  the  five  technologies  in  question  were  in 
development  for  several  years  with  development  participation  by  the  planned 
user.  The  Navy  needs  to  consider  whether  continued  expenditure  of  ATD  funds 
on  technologies  scheduled  for  near-tenn  transition  to  acquisition  programs  is 
prudent  if  the  necessary  funds  to  continue  the  required  development  are  not 
available. 

Although  the  Chief  of  Naval  Research  has  no  authority  to  compel  program 
managers  to  identify  transition  funding,  he  does  have  the  responsibility  to  review 
technology  paths  and  plans  for  technology  products  to  ensure  viability,  to  identify 
whether  the  necessary  transition  funds  are  available,  and  to  determine  whether 
continued  expenditure  of  ATD  funds  is  appropriate  when  technology  transitioning 
is  questionable.  With  limited  S&T  financial  resources,  it  is  prudent  to  align  S&T 
efforts  to  technologies  that  are  more  likely  to  result  in  successful  transitions.  The 
Handbook  and  Guide,  as  discussed  in  this  report,  emphasize  that  successful 
transitions  require  commitment  from  all  parties.  Coordination  and  commitment 
are  also  stressed  in  the  Defense  Systems  Management  College  training  course, 
“Program  Management  for  S&T  Manager”  STM  301,  formally  known  as 
“Technology  Insertion  in  Defense  Systems  Acquisitions.”  We  believe  that 
continued  expenditure  of  ATD  funds  on  technologies  that  do  not  have  coordinated 
paths  or  plans  for  transitioning  to  acquisition  programs  ignores  lessons  learned 
and  training  on  successful  S&T  transitioning  provided  to  Department  officials. 
Successful  transitioning  requires  coordination  and  funding  considerations  through 
formal  agreements  with  candidate  acquisition  program  managers.  The 
recommendation  was  not  intended  to  subject  ATDs  to  the  milestone  review 
process  used  for  acquisition  programs.  We  request  that  the  Navy  review  the 
technology  path  or  plans  for  the  five  unfunded  S&T  products,  determine  what 
action  should  be  taken  and  provide  additional  comments  to  the  final  report. 
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B.  Performance  Assessments 


The  performance  appraisal  process  was  not  effectively  used  as  a 
management  tool  to  assist  in  achieving  DoD  perfonnance  goals  and  the 
ONR  corporate  goals  of  transitioning  technology.  This  condition  exists 
because  the  ONR  did  not  incorporate  perfonnance  goals  necessary  for 
successful  technology  transitioning  into  the  S&T  product  managers’ 
performance  plans.  As  a  result,  the  ONR  did  not  fully  apply  the  Deputy 
Under  Secretary  of  Defense  for  Science  and  Technology’s  best  practices 
and  Navy  S&T  managers  were  not  held  accountable  for  transition  of 
technology. 

Background 


Deputy  Under  Secretary  of  Defense  for  Science  and  Technology.  The  Deputy 
Under  Secretary  of  Defense  for  Science  and  Technology  issued  a  Handbook  to  the 
Military  Departments  and  Defense  agencies  on  practices  that,  if  instituted,  would 
assist  in  achieving  the  DoD  goal  for  transitioning  technology.  ONR  established 
corporate  goals  to  be  used  as  a  perfonnance  benchmark  that  included  inserting 
technology  in  naval  operations. 

Addressing  Affordability  in  Defense  Science  and  Technology  (S&T): 

A  Handbook  for  S&T  Managers.  In  October  1999,  the  DoD  S&T  Affordability 
Task  Force  issued  a  Handbook  that  stresses  the  importance  of  involving  all 
candidate  acquisition  programs,  that  is,  the  acquisition  program  managers,  in 
developing  research.  As  described  in  the  Background  of  this  report,  the 
Handbook  emphasizes  close  coordination  with  technology  recipients.  The 
Handbook  also  states  that  one  of  the  keys  to  successful  transitioning  is 
implementing  an  S&T  personnel  assessment  process  that  is  based  on  transitioning 
and  affordability,  in  addition  to  individual  technical  achievements  and  publishing 
technical  papers. 

ONR  Corporate  Goals.  The  ONR  FY  2002  corporate  goals,  issued  on 
October  23,  2001,  continued  the  theme  of  goals  established  for  FY  2001.  The 
corporate  goals  established  the  basis  for  the  ONR  senior  executive  service 
objectives  and  performance  evaluation  and  provided  a  benchmark  for  establishing 
performance  objectives  and  assessments  for  all  personnel.  The  corporate  goals 
identified  four  broad  areas:  programmatic,  personnel,  financial,  and  business 
practices.  The  programmatic  area  includes  the  attribute  to  advance  development 
leading  to  products  that  transition  to  the  acquisition  community  for  insertion  into 
naval  operations. 

Performance  Plans 


The  personal  perfonnance  plans  for  the  ONR  science  and  engineering  senior 
executive  service  and  product  managers  did  not  include  adequate  S&T 
performance  requirements  necessary  to  enhance  technology  transitioning. 
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Examination  of  performance  plans  for  FNC  senior  executive  service  (known  as 
Level  4)  and  product  managers  (known  as  Level  3)  identified  critical  elements  in 
four  major  areas;  program  planning  and  leadership,  program  liaison  and 
teamwork,  program  execution,  and  supervision. 

Program  Planning  and  Leadership.  The  program  planning  and 
leadership  critical  perfonnance  element  for  senior  executive  service  and  product 
managers  included  three  broad  subcategories:  planning  scope  and  impact, 
program  innovation,  and  technical  credibility.  The  planning  scope  and  impact 
subcategory  focuses  on  providing  leadership  in  conceptualization  and  definition 
of  technical  areas  with  anticipated  Naval  needs.  Program  innovation  requires  the 
identification  of  promising  S&T,  programmatic,  or  organizational  concepts  of 
new  approaches.  Technical  credibility  means  being  recognized  as  an  expert  in  a 
technical  application  important  to  ONR. 

Program  Liaison  and  Teamwork.  The  senior  executive  service  and 
product  managers’  critical  perfonnance  element  for  program  liaison  and 
teamwork  included  the  categories  of  communication  and  interaction,  advocacy 
and  integration,  and  corporate  participation.  Communication  and  interaction 
involve  maintaining  communications  external  and  internal  to  DoD  on  product 
areas  of  ONR.  Advocacy  and  integration  focuses  on  collaborating  with  a  wide 
variety  of  S&T  experts  to  influence  investments  that  meet  ONR  product  areas  or 
strategic  goals.  Cooperative  participation  involves  fostering  successful  working 
relationships  and  consulting  with  higher-level  officials  on  complex  issues. 

Program  Execution.  The  senior  executive  service  and  product  managers’ 
critical  perfonnance  element  for  program  execution  had  two  subcategories:  asset 
management  and  program  assessment  and  documentation.  Asset  management 
includes  defining  near-term  and  long-term  asset  requirements  and  managing 
overall  program  assets  (people,  contracts,  funding,  and  equipment).  The  program 
assessment  and  documentation  subcategory  includes  defining  and  monitoring 
broad  corporate  metrics  for  measuring  program  success  in  meeting  ONR  needs, 
integrating  new  information  technologies  or  business  practices  to  achieve 
enhancement,  and  establishing  and  implementing  effective  processes  for 
monitoring  and  assessing  whether  the  quality  of  program  research  efforts  are 
consistent  with  the  ONR  corporate  metrics  framework. 

Supervision.  The  senior  executive  service  and  product  managers’  critical 
performance  element  for  supervision  had  three  subcategories:  organizational 
development,  performance  management,  and  workforce  development. 
Organizational  development  includes  establishing  an  infrastructure  for  multiple 
work  units  or  teams  to  achieve  mission  and  program  goals,  promoting  best 
practices  in  organization  development,  and  attracting  high  performing  scientists 
and  engineers.  Performance  management  involves  establishing  organizational 
goals  and  objectives,  internal  controls,  and  perfonnance  management  and 
incentive  techniques  that  monitor  achievements  and  provide  recognition.  The 
workforce  development  subcategory  includes  mentoring  for  career  growth, 
developing  successful  technical  and  leadership  team  members,  encouraging 
continuous  learning,  and  coaching  subordinates. 
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The  inclusion  of  program  planning  and  leadership,  program  liaison  and  teamwork, 
program  execution,  and  supervision  are  all  important  performance  elements  for 
the  product  managers;  however,  the  perfonnance  elements  did  not  emphasize  the 
importance  of  advanced  technology  development  transitioning.  The  inclusion  of 
a  performance  element  for  technology  transitioning  and  the  attributes  for 
coordination  and  technology  transitioning  identified  in  the  Deputy  Under 
Secretary  of  Defense  for  Science  and  Technology  Handbook  and  Guide  would 
help  highlight  that  proven  technology  must  transition  to  the  warfighter. 

Although  the  ONR  performance  plans  provide  for  tailoring  performance 
expectations  to  address  specific  technologies  that  product  managers  are 
responsible  for,  according  to  ONR  officials,  none  of  the  performance  plans  were 
tailored  to  include  expected  technical  accomplishments  during  the  performance 
period.  The  perfonnance  standards  did  address  achieving  ONR  goals,  but  such  a 
generic  standard  lacks  the  specificity  and  the  ability  to  measure  achievement  of 
goals  and  measures  at  the  individual  level.  The  incorporation  of  specific, 
expected  product  achievements  such  as  reaching  a  technical  milestone,  meeting  a 
product  timeframe  schedule,  establishing  and  using  working-level  IPTs  with  all 
planned  users  (as  opposed  to  only  at  the  FNC  level),  or  achieving  a  technology 
transition  to  the  acquisition  community  would  be  a  better  measure  of  product 
managers’  perfonnance.  The  inclusion  or  referencing  of  expected  perfonnance 
elements  would  assist  in  achieving  the  ONR  corporate  goals  by  providing  a  link 
between  performance  assessments  and  technology  transitioning  and  would  hold 
Navy  S&T  managers  accountable  for  transition  performance. 

Recommendation,  Management  Comment,  and  Audit 
Response 


B.  We  recommend  that  the  Chief  of  Naval  Research  require  that  the 
personnel  performance  plans  for  product  managers  responsible  for  advanced 
technology  development-funded  programs  explicitly  require  a  supervisor’s 
assessment  of  the  manager’s  performance  with  planned  technology  users. 

The  performance  plans  should  include  performance  assessments  for 
establishment  of  working-level  integrated  product  teams  with  all  planned 
(primary  and  secondary)  technology  users,  creation  of  integrated  product 
team  charters,  coordination  and  acceptance  of  quantitative  metrics  and  key 
exit  criteria  with  all  planned  users,  development  of  transition  plans  that  are 
formally  agreed  to  by  all  planned  users,  and  development  and  maintenance 
of  up-to-date  memorandums  of  agreement  or  understanding. 

Management  Comments.  The  Deputy  Assistant  Secretary  concurred  and  stated 
that  ONR  is  revising  its  personnel  performance  system  with  planned 
implementation  in  FY  2004  to  contain  a  critical  element  for  Program  Planning 
and  Leadership.  The  Program  Planning  and  Leadership  perfonnance  element 
would  require  “working  results  in  significant  new  scientific  and/or  technical 
results  as  well  as  applications  or  other  outcomes  that  significantly  expand  the 
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future  capabilities  of  the  Navy  and  Marine  Corps.”  The  Deputy  Assistant 
Secretary  stated,  in  addition,  that  ONR  has  begun  a  database  of  organizational 
metrics,  which  includes  transitions  as  a  major  indicator  of  success. 

Audit  Response.  The  Deputy  Assistant  Secretary’s  comments  are  not  fully 
responsive  to  the  recommendation.  The  recommendation  requires  that  technology 
transition  be  identified  as  a  performance  element,  with  specific  actions  to  be  used 
as  measures  of  product  managers’  performance  to  enhance  technology  transitions. 
The  Army  has  agreed  to  add  technology  transition  as  an  element  to  the  program 
managers’  perfonnance  plans  and  the  Air  Force  performance  plans  specifically 
addresses  technology  transition  as  an  element.  However  the  Navy  comments 
describe  a  performance  system  that  is  generic  and  does  not  address  the  specific 
levels  of  performance  from  product  managers,  such  as  establishing  IPTs, 
coordinating  quantitative  metrics  and  key  exit  criteria,  developing  transition 
plans,  and  maintaining  current  memorandums  of  agreement  or  understanding.  We 
believe  the  inclusion  of  these  specific  elements  in  performance  plans  are 
necessary  to  enhance  the  likelihood  of  technology  transitions.  We  request 
additional  management  comments  to  the  final  report  that  provide  more  definitive 
actions  related  to  the  recommendation. 
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C.  Effectiveness  of  the  Management 
Control  Program 

The  ONR  did  not  effectively  use  the  management  control  program  as  a 
management  tool  for  evaluating  the  technology  transition  operations 
within  the  S&T  Directorate.  This  condition  occurred  because  ONR  did 
not  conduct  periodic  command  reviews  to  assess  compliance  with 
management  control  policy,  ONR  managers  did  not  receive  management 
control  training  consistent  with  their  responsibilities  and  obligations,  and 
ONR  did  not  identify  sufficient  management  controls  to  assess  the 
transition  of  technologies.  As  a  result,  the  ONR  annual  statements  of 
assurance  are  not  based  on  reliable  assessments  of  the  S&T  process  and 
did  not  adequately  assess  risks  and  controls. 

Management  Control  Criteria 


The  Office  of  Management  and  Budget  Circular  A- 123,  Revised,  “Management 
Accountability  and  Control,”  June  21,  1995,  requires  agencies  to  adequately 
document  management  control  procedures  and  states: 

Agencies  and  individual  Federal  managers  must  take  systematic  and  proactive  measures 
to  (i)  develop  and  implement  appropriate,  cost-effective  management  controls  for  results- 
oriented  management;  (ii)  assess  the  adequacy  of  management  controls  in  Federal 
programs  and  operations;  (iii)  identify  needed  improvements;  (iv)  take  corresponding 
corrective  action;  and  (v)  report  annually  on  management  controls. 

DoD  Directive  5010.38,  “Internal  Management  Control  Program,”  August  26, 
1996,  requires  DoD  organizations  to  implement  a  comprehensive  strategy  for 
management  controls  that  provides  reasonable  assurance  that  programs  and 
administrative  and  operating  functions  are  efficiently  and  effectively  carried  out 
in  accordance  with  applicable  law  and  management  policy.  The  directive  requires 
continuous  monitoring  and  periodic  evaluations  that  provide  the  basis  for  the 
annual  statement  about  reasonable  assurance.  Management  control  objectives 
include  executing  functions  properly;  avoiding  fraud,  waste,  and  mismanagement; 
safeguarding  assets;  accounting  for  revenues  and  expenditures  properly;  and 
complying  with  laws  and  regulations. 

DoD  Instruction  5010.40,  “Management  Control  Program  Procedures,” 

August  28,  1996,  implements  policy,  assigns  responsibilities,  and  prescribes 
procedures  for  management  control  programs.  The  Instruction  provides  guidance 
for  determining  materiality  for  management  control  weaknesses. 

SECNAV  Instruction  5200. 35D,  “Department  of  the  Navy  Management  Control 
Program,”  December  10,  1997,  requires  that  all  Department  of  the  Navy 
Components  maintain  effective  management  control  systems  and  continually 
monitor  and  improve  the  effectiveness  of  controls  chosen  to  be  employed  for  their 
programs.  Monitoring  efforts  should  be  documented  and  performance  appraisal 
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systems  should  reflect  any  management  control  responsibilities, 
accomplishments,  deficiencies,  and  corrective  actions  taken.  Each  Department  of 
the  Navy  Component  should  establish  criteria  for  scheduling  evaluations  of 
assessable  units. 

ONR  Instruction  5200.  IB,  “Office  of  Naval  Research  Management  Control 
Program,”  June  8,  2001,  requires  that  all  ONR  Components  maintain  effective 
management  control  systems,  and  that  all  levels  of  ONR  management  establish 
controls,  and  the  effectiveness  of  controls  established  for  programs  be  continually 
monitored  and  improved.  The  Instruction  assigns  responsibility  for  an  annual 
review  of  one -third  of  the  corporate  staff  and  departments  to  the  Corporate 
Business  Council. 

ONR  Management  Controls 


Management  controls  need  to  be  adequately  designed,  documented,  and  tested  to 
provide  assurance  that  command  resources  are  used  efficiently  and  effectively. 
The  S&T  departments  did  not  identify  sufficient  management  controls  to  meet  the 
objectives  of  the  management  control  process.  In  some  instances,  controls  were 
not  documented  and,  in  others,  controls  were  documented  but  could  not  be 
quantified. 

Management  Controls  Within  the  S&T  Directorate.  The  ONR  S&T 

Directorate  comprises  six  departments;  five  departments  are  responsible  for 
technology  development,  which  is  the  focus  of  this  review.  Three  of  the  five  S&T 
departments  used  the  same  management  control  plan.  All  five  S&T  departments 
provided  statements  in  FY  2001  that  they  had  reasonable  assurance  that 
management  controls  were  in  place  and  operating  effectively.  Further 
examination  disclosed  that  two  of  the  S&T  departments  could  not  provide 
documentation  to  identity  the  management  controls  that  they  had  monitored  or 
tested.  The  three  remaining  S&T  departments  had  documented  management 
control  plans  used  to  derive  their  management  control  annual  assessments. 

The  standard  management  control  plan  is  composed  of  four  assessable  units:  plan 
and  execute  S&T  programs,  lead  and  leverage  other  S&T  for  the  Department  of 
the  Navy,  customer  relations  and  transitions,  and  administrative  support.  The 
four  assessable  units  have  subcategories  called  controls  that  are  used  to  measure 
management’s  effectiveness  in  managing  the  assessable  units.  The  controls 
identified  by  the  S&T  departments  are  provided,  verbatim,  in  the  table  that 
follows. 
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ONR  S&T  Management  Control  Plan 


Assessable  unit 

Management  Control 

How  Measured 

Plan/Execute  Science 

a)  Attract  best  Principal  Investigators  for  task 

Judgment 

and  Technology 

b)  Ensure  quality/innovation  and  state-of-the-art 

Judgment 

Programs 

c)  Balance  of  basic-applied-demo  and  risk 

Judgment 

d)  Ensure  naval  relevance 

Judgment 

e)  Properly  integrated/coordinated 

Judgment 

Lead/Leverage  Other 

a)  Attract  best  Program  Officers  and 

Judgment 

Science  and 

sustain/grow  skills 

Technology  for 

b)  Technological  awareness  at  forefront(s) 

Judgment 

Department  of  the 

c)  Interagency  activities  as  leader 

Judgment 

Navy 

d)  International  activities  as  leader 

Judgment 

Customer  Relations 

a)  Requirements  and  needs  interactions 

Judgment 

and  Transitions 

b)  Extrapolations  based  on  science  and 
technology  potentials 

Judgment 

c)  Work  the  transitions 

Judgment 

d)  Marketing  of  science  and  technology  "value- 
added" 

Judgment 

Administrative 

a)  Inquiries  (Congress,  Freedom  of  Information 

Review  submittals 

Support  Activities 

Act,  etc.) 

b)  Personnel  actions  and  activities 

Judgment 

c)  Timekeeping 

Review  submittals 

d)  Science  and  technology  awards  process 

Review  submittals 

e)  Travel 

Review  submittals 

f)  Fiscal  tracking 

Review  submittals 

g)  Management  information  systems 

Review  submittals 

Why  Management  Controls  Need  Improvement 


Need  for  Periodic  Reviews  by  the  Corporate  Business  Council.  ONR 

Instruction  5200.  IB,  “Office  of  Naval  Research  Management  Control  Program,” 
June  8,  2001,  assigns  responsibility  for  reviewing  compliance  with  management 
control  policy  to  its  Corporate  Business  Council.  The  Corporate  Business 
Council  is  required  to  review  one-third  of  the  corporate  staff  and  departments 
each  year.  The  Corporate  Business  Council  did  not  conduct  any  command 
reviews.  If  the  Corporate  Business  Council  had  conducted  reviews  as  required,  it 
may  have  identified  weaknesses  in  the  ONR  management  control  program.  In 
addition,  thorough  reviews  would  have  revealed  that  management  controls  were 
neither  specific  nor  measurable. 
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Appropriate  Management  Control  Training  for  Key  S&T  Managers.  ONR 

did  not  comply  with  DoD  Directive  5010.38  and  SECNAV  Instruction  5200. 35D 
requirements  to  provide  training  to  managers,  consistent  with  their  management 
control  responsibilities.  An  ONR  management  control  official  stated  that  each 
S&T  department  head  was  provided  a  briefing  on  management  controls. 

However,  the  department  heads  were  not  provided  formal  training.  ONR 
established  an  Internet  web  site  for  management  control  training,  but  the  site 
serves  more  as  a  resource  center  than  a  training  venue  because  personnel  are 
allowed  to  self-certify  training  completion  without  demonstrating  any 
comprehension  of  management  controls. 

Management  Controls  for  Transition  of  S&T  Technologies.  ONR  identified 
just  one  management  control  that  was  specific  to  the  transition  of  technologies. 

As  shown  in  the  preceding  table  of  S&T  management  controls,  ONR  identified 
the  technology  transition  effort  within  assessable  unit  number  3.,  “Customer 
Relations  and  Transitions.”  The  control  identified  with  the  technology  transition 
effort  is  “Work  the  transition.”  The  assessment  of  this  one  identifiable  control 
over  technology  transitions  is  completed  by  the  judgment  of  the  S&T  officials. 
ONR  can  improve  the  management  controls  over  technology  transition  by 
identifying  additional  controls  that  are  quantifiable  and  measurable,  such  as: 

•  Establishing  and  keeping  current  memorandums  of  agreement  or 
understanding,  or  technology  transition  agreements  that  include  TRLs,  exit 
criteria,  transition  timelines,  and  funding  agreements  for  development  and 
transitioning. 

•  Establishing  and  maintaining  a  working-level  IPT  charter  that  identifies 
the  roles  and  responsibilities  of  the  participating  parties. 

•  Establishing  a  working-level  IPT  with  the  acquisition  program  manager 
and,  as  appropriate,  the  prime  contractor. 

•  Recording,  coordinating,  and  establishing  agreement  on  the  decisions, 
issues,  and  action  items  resulting  from  the  working-level  IPT  meetings. 

•  Coordinating  secondary  transition  paths. 

Effect  of  Weaknesses  Noted 


Until  adequate  management  controls  are  identified  and  tested,  ONR  cannot 
determine  with  reasonable  assurance  that  the  S&T  Directorate  is  operating 
efficiently  and  effectively. 


21 


Recommendations,  Management  Comments,  and  Audit 
Response 


C.  We  recommend  that  the  Chief  of  Naval  Research, 

1.  Require  Science  &  Technology  Department  officials  responsible  for 
technology  transition  programs  to  identify  and  document  appropriate 
management  controls  that  are  measurable  and  that  will  contribute  to 
preparing  the  annual  statement  of  assurance. 

Management  Comments.  The  Deputy  Assistant  Secretary  concurred  and  stated 
that  ONR  formed  an  S&T  Performance  Measurement  Analysis  Program, 
established  performance  measurement  prototypes,  convened  a  Working  Group, 
conducted  a  study  of  vendors,  and  developed  three  web-based  perfonnance 
measurement  prototypes  for  test  and  evaluation.  Preliminary  data  had  been 
collected  and  one  of  the  metrics  is  transitioning. 

Audit  Response.  Although  the  Deputy  Assistant  Secretary  concurred  the 
comments  do  not  address  the  intent  of  the  recommendation  to  establish 
quantifiable  and  measurable  assessable  units.  It  is  unclear  how  the  processes 
described  in  the  response  affect  the  ONR  S&T  management  controls  that  we 
reviewed.  The  table  on  page  20  represents  the  ONR  S&T  official  management 
control  program.  The  comments  do  not  address  the  finding,  the  table,  or  the 
recommendation  to  modify  the  S&T  management  control  program  to  include 
measurable  controls.  We  request  that  ONR  officials  review  the  suggested 
improvements  to  the  S&T  management  control  program  as  discussed  on  page  2 1 
(establishing  memorandums  of  agreement,  technology  transition  agreements, 
establishing  working-level  IPT  charters)  and  provide  additional  comments  that 
address  how  the  management  control  program,  with  measurable  controls,  will  be 
used  as  a  management  tool  to  enhance  technology  transitioning. 

2.  Direct  the  Corporate  Business  Council  to  conduct  annual 
command  reviews  and  use  the  review  results  as  a  basis  for  the  Office  of  Naval 
Research  annual  statement  of  assurance. 

Management  Comments.  The  Deputy  Assistant  Secretary  concurred  and  stated 
that  the  ONR  Corporate  Business  Council  reviewed  59  of  138  ONR  assessable 
units.  The  review  results  contributed  to  the  ONR  FY  2002  annual  assurance 
statement.  The  reviews  will  continue  contributing  to  future  statements. 
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3.  Provide  management  control  training  to  the  Science  and 
Technology  officials,  consistent  with  the  requirements  of  DoD 
Directive  5010.38. 

Management  Comments.  The  Deputy  Assistant  Secretary  concurred  and  stated 
that  ONR  will  continue  to  provide  training  to  managers  consistent  with  their 
responsibilities  and  obligations.  ONR  developed  web-based  management  control 
training  in  recognition  of  the  need  to  have  a  means  of  delivery  available  on 
demand.  The  Deputy  Assistant  Secretary  stated  that  the  web-based  training  is 
augmented  by  assistance  from  the  Management  Control  Coordinator  when 
needed,  required,  or  requested. 

Audit  Response.  The  Deputy  Assistant  Secretary  comments  did  not  address  the 
intent  of  the  recommendation.  We  were  aware  that  the  S&T  officials  received 
personal  briefings  and,  in  some  cases,  completed  the  web-based  management 
control  training.  The  comments  restate  the  training  program  that  existed  during 
the  audit.  Our  discussions  with  S&T  officials  concerning  the  management  control 
program  identified  that  the  officials’  comprehension  of  management  controls  was 
not  consistent  with  their  responsibilities  and  obligations,  thus  resulting  in  this 
finding  and  recommendation.  For  example,  one  ONR  official  requested  the 
auditors  to  explain  an  assessable  unit  prior  to  discussing  the  management  control 
plan.  This  official  reported  a  high  confidence  in  controls  in  his  annual  report  for 
this  assessable  unit.  In  addition,  only  two  of  six  S&T  Department  officials  had 
completed  the  web-based  management  control  training.  Management  comments 
do  not  identify  any  additional  actions  to  be  taken  as  a  result  of  this  finding  that 
would  enhance  the  S&T  officials’  understanding  of  management  controls.  We 
request  that  the  Navy  provide  additional  comments  to  the  final  report  discussing 
how  management  control  training  will  be  improved  to  ensure  that  S&T  officials 
understand  the  program  and  their  responsibilities  and  can  use  this  knowledge  to 
develop  quantifiable  and  measurable  controls. 
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Appendix  A.  Scope  and  Methodology 


Work  Performed.  We  examined  39  S&T  technologies  at  the  ONR  to  evaluate 
the  management  process  for  transitioning  successful  technologies  to  the 
warfighter.  The  S&T  technologies  examined  were  funded  with  the  advanced 
technology  development  budget  activity  within  the  Navy  research,  development, 
test  and  evaluation  appropriations,  and  had  a  FY  2002  budget  of  $870.3  million, 
which  includes  $189.8  million  in  congressional  increases.  The  FY  2003  budget 
request  is  $617.1  million. 

We  conducted  interviews  with  S&T  and  acquisition  program  officials,  and 
examined  applicable  key  documentation.  Key  documentation  included  guidance 
advocated  by  the  Deputy  Under  Secretary  of  Defense  for  Science  and 
Technology;  FNC  business  plans;  technology  transition  agreements;  IPT  meeting 
minutes  where  available;  memorandums  of  understanding  and  agreement; 
acquisition  program  funding  profiles;  S&T  management  plans;  technology 
transition  paths  or  plans;  and  the  research,  development,  test,  and  evaluation 
budget  item  justification  sheet  (R-2  Exhibit). 

We  performed  this  audit  from  January  2002  through  August  2002  in  accordance 
with  generally  accepted  government  auditing  standards  with  the  exception  of  the 
scope  limitations  noted  below. 

Limitations  to  Scope.  We  did  not  evaluate  the  technical  merits  of  the  S&T 
products.  We  limited  the  ONR  management  control  review  to  the  S&T 
management  procedures  and  the  departments  responsible  for  transitioning 
technology  from  S&T  community  to  the  acquisition  program  managers  or  another 
technology  area. 

Use  of  Computer-Processed  Data.  We  relied  on  computer-processed  data 
without  performing  tests  of  general  and  application  system  controls  to  confirm  the 
reliability  of  the  data.  We  relied  on  the  Department  of  the  Navy  Science  and 
Technology  Program  website,  https://donst.nrl.navy.mil/donst/,  and  the 
computerized  listing  to  represent  the  known  universe  of  S&T  products  in  the 
review  of  the  management  process.  We  validated  the  total  funding  for  FY  2002 
on  the  computerized  listing  to  the  total  funding  under  the  advanced  technology 
development  budget  activity  within  the  Navy  research,  development,  test,  and 
evaluation  appropriation.  Validating  the  computerized  listing  to  the 
appropriations  was  appropriate  for  this  audit  because  the  audit’s  objective  was  to 
examine  the  management  process  for  transitioning  technology,  not  the  individual 
S&T  products.  Further  validation  of  the  computerized  listing  would  not  change 
the  conclusions  in  this  report. 

General  Accounting  Office  High-Risk  Area.  The  General  Accounting  Office 
has  identified  several  high-risk  areas  in  the  DoD.  This  report  provides  coverage 
of  the  Weapon  System  Acquisition  high-risk  area. 
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Management  Control  Program  Review 


DoD  Directive  5010.38,  “Management  Control  (MC)  Program,”  August  26,  1996, 
and  DoD  Instruction  5010.40,  “Management  Control  (MC)  Program  Procedures,” 
August  28,  1996,  require  DoD  organizations  to  implement  a  comprehensive 
system  of  management  controls  that  provides  reasonable  assurance  that  programs 
are  operating  as  intended  and  to  evaluate  the  adequacy  of  the  controls. 

SECNAV  Instruction  5200. 35D,  “Department  of  the  Navy  Management  Control 
Program,”  December  10,  1997,  requires  that  all  Department  of  the  Navy 
Components  maintain  effective  management  control  systems  and  continually 
monitor  and  improve  the  effectiveness  of  controls  chosen  to  be  employed  for  their 
programs.  ONR  Instruction  5200.  IB,  “Office  of  Naval  Research  Management 
Control  Program,”  June  8,  2001,  requires  that  all  ONR  Components  maintain 
effective  management  control  systems. 

Scope  of  the  Review  of  the  Management  Control  Program.  We  evaluated  the 
management  control  process  at  ONR  to  determine  whether  effective  management 
procedures  were  established  to  transition  successful  S&T  projects  to  acquisition 
program  managers  and,  ultimately,  to  the  warfighter. 

Adequacy  of  Management  Controls.  We  identified  a  material  management 
control  weakness  for  ONR  as  defined  by  DoD  Instruction  5010.40.  ONR 
management  controls  for  technology  transition  within  the  S&T  Directorate  were 
not  adequate  to  provide  reasonable  assurance  that  the  transition  effort  had  been 
executed  efficiently  and  effectively.  Recommendations  C.I.,  C.2.,  and  C.3.,  if 
implemented,  will  improve  ONR  management  controls  over  the  technology 
transition  effort.  A  copy  of  the  report  will  be  provided  to  the  Navy  official 
responsible  for  management  controls. 

Adequacy  of  Management’s  Self-Evaluation.  ONR  officials  identified 
technology  transition  with  the  S&T  Directorate  as  part  of  an  assessable  unit. 
However,  in  its  evaluation,  ONR  officials  did  not  identify  the  specific  material 
management  control  weaknesses  identified  by  the  audit  because  the  ONR 
evaluation  was  too  general  and  not  measurable. 

Prior  Coverage 


During  the  last  5  years,  the  General  Accounting  Office  has  issued  one  report  and 
the  Inspector  General  of  the  Department  of  Defense  has  issued  two  reports 
discussing  the  benefits  of  adequately  managing  the  challenges  of  transitioning 
technologies  to  warfighters. 

General  Accounting  Office 

Report  No.  NSIAD-99-162,  “Best  Practices:  Better  Management  of  Technology 
Development  Can  Improve  System  Outcomes,”  July  30,  1999 
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Inspector  General  of  the  Department  of  Defense  (IG  DoD) 

IG  DoD  Report  No.  D-2002-107,  “Army  Transition  of  Advanced  Technology 
Programs  to  Military  Applications,”  June  14,  2002 

IG  DoD  Report  No.  D-2002-146,  “The  Defense  Advanced  Research  Projects 
Agency’s  Transition  of  Advanced  Information  Technology  Programs,” 
September  11,  2002 
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Appendix  B.  Technology  Readiness  Levels  and 

Their  Definitions 


The  Deputy  Secretary  of  Defense  issued  interim  acquisition  guidance  on  October  30, 
2002.  The  interim  guidance  provided  a  matrix  that  lists  technology  readiness  levels  and 
descriptions  from  a  systems  approach  for  both  hardware  and  software  as  shown  below. 


Technology  Readiness  Level 

Description 

1.  Basic  principles  observed  and 
reported. 

Lowest  level  of  technology  readiness.  Scientific  research  begins  to 
be  translated  into  applied  research  and  development.  Examples 
might  include  paper  studies  of  a  technology’s  basic  properties. 

2.  Technology  concept  and/or 
application  formulated. 

Invention  begins.  Once  basic  principles  are  observed,  practical 
applications  can  be  invented.  The  applications  are  speculative  and 
there  may  be  no  proof  or  detailed  analysis  to  support  the 
assumptions.  Examples  are  limited  to  analytic  paper  studies. 

3.  Analytical  and  experimental 
critical  function  and/or  characteristic 
proof  of  concept. 

Active  research  and  development  is  initiated.  This  includes 
analytical  studies  and  laboratory  studies  to  physically  validate 
analytical  predictions  of  separate  elements  of  the  technology. 
Examples  include  components  that  are  not  yet  integrated  or 
representative. 

4.  Component  and/or  breadboard 
validation  in  laboratory 
environment. 

Basic  technological  components  are  integrated  to  establish  that  they 
will  work  together.  This  is  relatively  “low  fidelity”  compared  to  the 
eventual  system.  Examples  include  integration  of  “ad  hoc” 
hardware  in  a  laboratory. 

5.  Component  and/or  breadboard 
validation  in  relevant  environment. 

Fidelity  of  breadboard  technology  increases  significantly.  The 
basic  technological  components  are  integrated  with  reasonably 
realistic  supporting  elements  so  it  can  be  tested  in  simulated 
environment.  Examples  include  “high  fidelity”  laboratory 
integration  of  components. 

6.  Sy stem/ subsystem  model  or 
prototype  demonstration  in  a 
relevant  environment. 

Representative  model  or  prototype  system,  which  is  well  beyond 
that  of  TRL  5,  is  tested  in  a  relevant  environment.  Represents  a 
major  step  up  in  a  technology’s  demonstrated  readiness.  Examples 
include  testing  a  prototype  in  a  high  fidelity  laboratory  environment 
or  in  a  simulated  operational  environment. 

7.  System  prototype  demonstration 
in  an  operational  environment. 

Prototype  near,  or  at,  planned  operational  system.  Represents  a 
major  step  up  from  TRL  6,  requiring  the  demonstration  of  an  actual 
system  prototype  in  an  operational  environment  such  as  an  aircraft, 
vehicle,  or  space.  Examples  include  testing  the  prototype  in  a  test 
bed  aircraft. 

8.  Actual  system  completed  and 
qualified  through  test  and 
demonstration. 

Technology  has  been  proven  to  work  in  its  final  form  and  under 
expected  conditions.  In  almost  all  cases,  this  TRL  represents  the 
end  of  true  system  development.  Examples  include  developmental 
test  and  evaluation  of  the  system  in  its  intended  weapon  system  to 
determine  if  it  meets  design  specifications. 

9.  Actual  system  proven  through 
successful  mission  operations. 

Actual  application  of  the  technology  in  its  final  form  and  under 
mission  conditions,  such  as  those  encountered  in  operational  test 
and  evaluation.  Examples  include  using  the  system  under 
operational  mission  conditions. 
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Knowledge  Superiority  and  Assurance 
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Note:  See  footnotes  at  the  end  of  the  appendix. 
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Note:  See  footnotes  at  the  end  of  the  appendix. 


Knowledge  Superiority  and  Assurance  (cont’d) 
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Note:  See  footnotes  at  the  end  of  the  appendix. 


Knowledge  Superiority  and  Assurance  (cont’d) 
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Note:  See  footnotes  at  the  end  of  the  appendix. 


Littoral  Anti-Submarine  Warfare 


Note:  See  footnotes  at  the  end  of  the  appendix. 


Organic  Mine  Countermeasures 
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■:  See  footnotes  at  the  end  of  the  appendix. 


Fleet  Force  Protection 
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Note:  See  footnotes  at  the  end  of  the  appendix. 
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Note:  See  footnotes  at  the  end  of  the  appendix. 


Fleet  Force  Protection  (cont’d) 


36 


Note:  See  footnotes  at  the  end  of  the  appendix. 


Advanced  Technology  Demonstrations 
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Note:  See  footnotes  at  the  end  of  the  appendix. 


Advanced  Technology  Demonstrations  (cont’d) 
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Appendix  D.  Report  Distribution 


Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 
Deputy  Under  Secretary  of  Defense  for  Science  and  Technology 
Under  Secretary  of  Defense  (Comp troller)/Chief  Financial  Officer 
Deputy  Chief  Financial  Officer 
Deputy  Comptroller  (Program/Budget) 


Department  of  the  Navy 

Assistant  Secretary  of  the  Navy  (Research,  Development,  Acquisition) 
Chief  of  Naval  Research 

Direct  Reporting  Program  Manager,  Advanced  Amphibious  Assault 
Commander,  Naval  Air  Systems  Command 
Commander,  Naval  Sea  Systems  Command 
Commander,  Space  and  Naval  Warfare  Systems  Command 
Commander,  Marine  Corps  Systems  Command 
Naval  Inspector  General 
Auditor  General,  Department  of  the  Navy 


Non-Defense  Federal  Organization 

Office  of  Management  and  Budget 


Congressional  Committees  and  Subcommittees,  Chairman  and 
Ranking  Minority  Member 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 
Senate  Committee  on  Anned  Services 
Senate  Committee  on  Governmental  Affairs 
House  Committee  on  Appropriations 

House  Subcommittee  on  Defense,  Committee  on  Appropriations 
House  Committee  on  Armed  Services 
House  Committee  on  Government  Reform 

House  Subcommittee  on  Government  Efficiency,  Financial  Management,  and 
Intergovernmental  Relations,  Committee  on  Government  Reform 
House  Subcommittee  on  National  Security,  Veterans  Affairs,  and  International  Relations, 
Committee  on  Government  Reform 

House  Subcommittee  on  Technology  and  Procurement  Policy,  Committee  on 
Government  Reform 
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Department  of  the  Navy  Comments 


DEPARTMENT  OF  THE  NAVY 
OFFICE  OF  THE  ASSISTANT  SECRETARY 
RESEARCH.  DEVELOPMENT  AMD  ACQUISITION 
1000  NAVY  PENTAGON 
WASHINGTON.  OC  203SO-1000 


20  November  2002 


MEMORANDUM  FOR  THE  DEPARTMENT  OF  DEFENSE  ASSISTANT  INSPECTOR  GENERAL 
FOR  AUDITING 

Subj  :  DRAFT  AUDIT  REPORT  ON  NAVY  TRANSITION  OF  ADVANCED  TECHNOLOGY  PROGRAMS 
TO  MILITARY  APPLICATIONS  (PROJECT  NO.  D200IAB- 0105. 001) 

Ref:  (a)  DoDIG  memo  of  30  Sept  02 

Enel:  (1)  Department  of  the  Navy  Response 

In  response  to  your  memorandum  of  30  September  2002,  the  Department  of 
the  Navy  appreciates  the  opportunity  to  provide  comments  on  the  subject 
report.  The  enclosure  provides  concurrence  or  non- concurrence  with  each 
reconraendation,  and  coimnents  as  you  requested. 

I  wish  to  emphasize  the  Navy’s  disagreement  with  two  of  the  report's 
recommendations,  which  could  have  an  adverse  impact  on  the  Navy’s  ability  to 
transition  new  technology  to  our  operating  forces.  Contrary  to  Under 
Secretary  of  Defense  (Acquisition,  Technology  and  Logistics)  policy,  these 
recoraaendat ions  are  overly  prescriptive,  and  are  harmful  to  an  acquisition 
policy  environment  that  fosters  efficiency,  creativity,  and  innovation. 

Recommendation  A.l.  suggests  that  the  Chief  of  Naval  Research  require 
inclusion  of  the  acquisition  program  office's  prime  contractors,  where 
applicable,  in  integrated  product  teams  IIPT)  established  for  advanced 
technology  development  (6,3)  efforts.  All  Navy  6.3  efforts  have,  or  soon 
will  have,  IPTs  including  representatives  of  targeted  acquisition  programs. 
The  acquisition  program  manager  should  have  sole  authority  to  decide  whether, 
when,  and  how  hie  prime  contractors  participate  in  these  IPTs.  In  practice, 
most  6.3  IPTs  will  include  the  prime  contractors  when  appropriate.  Requiring 
the  inclusion  of  prime  contractors  in  all  cases  would  leave  the  government 
open  for  potential  problems.  Including  the  appearance  of  pre-selection  for 
future  solicitations  and  possible  abuses  of  proprietary  information. 

Recommendation  A. 3.  requires  that  the  chief  of  Naval  Research 
discontinue  any  product  development  for  6.3  efforts  scheduled  to  transition 
in  PY  2002-2003  that  does  not  have  identified  funding  from  an  acquisition 
program.  This  recommendation,  if  implemented,  would  limit  6.3  funding  to 
projects  that  had  already  passed  acquisition  milestone  A,  and  thereby 
forestall  nearly  all  transformational  technology  development .  It  is 
essential  that  6.3  programs  be  able  to  demonstrate  unproven  options,  mitigate 
risks,  and  offer  alternative  solutions  before  an  acquisition  program  conmits 
to  transition  the  product. 

UUdJLlJL4~ 

WILLIAM  J.  SCHAEFER 
Deputy  Assistant  Secretary  of 
the  Navy 

Planning,  Programming  and 
Resources 
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Department  of  the  Navy  Response 
to 

AIG(A)  Draft  Report  of  30  September  2002 
on 

Navy  Transition  of  Advanced  Technology  Programs  to  Military 
Applications  (Project  No.  D2001AB-0105.001) 


Recommendation  A.l: 

We  recommend  that  the  Chief  of  Naval  Research,  for  advanced 
technology  development  efforts.-  1.  Require  the  establishment  of 
integrated  product  teams  that  include  representatives  from  the 
primary  and  secondary  candidate  acquisition  program  office (s)  and 
the  acquisition  program  office  prime  contractor (si  where 
applicable. 

Department  of  Che  Navy  Response: 

Partially  concur.  Recommend  removal  of  the  phrase  "and  the 
acquisition  program  office  prime  contractor (s)  where  applicable"  or 
substitution  of  the  alternative  phrase  "with  the  acquisition  office's 
prime  contractor (s)  when  desired  by  the  acquisition  program  manager." 

The  Navy  is  implementing  the  first  part  of  the  recommendation  to 
require  establishment  of  integrated  product  teams  (IPTs)  that  include 
target  acquisition  program  offices  as  members.  The  Navy  has  divided 
advanced  technology  programs  into  two  groups.  Future  Naval 
Capabilities  (FNCs)  and  Exploitation  and  Development  (BAD) .  FNCs 
provide  S&T  support  to  near  and  mid  term  warfighter  requirements  and 
acquisition  programs,  with  focus  on  transition  within  five  years.  E&D 
programs  support  higher  risk  transformational  efforts. 

Subject  audit  was  conducted  in  FY  2002,  the  first  year  of 
implementation  of  the  FNCs.  All  PNC  programs  are  required  to  have 
IPTs  as  recommended.  At  present,  all  FNCs  have  IPTs  at  the  program 
level,  and  are  forming  IPTs  for  transition  of  each  project.  For 
transformational  E&D  efforts,  an  IPT  will  be  the  norm.  It  is 
imprudent  to  require  IPTs,  or  any  bureaucratic  instrument,  in  every 
case;  but  exceptions  should  and  will  be  rare,  justified,  and 
documented.  FNCs  will  document  compliance  with  this  portion  of  the 
recommendation  in  the  ONR  Investment  Balance  Review  in  January  2003, 
and  management  control  statements  of  assurance  for  FY  2003. 

The  recommendation  that  the  CNR  require  inclusion  of  prime 
contractor (s)  in  the  IPTs  is  not  executable  and  potentially  harmful  to 
the  acquisition  of  new  technologies.  IPTs  for  6.3  programs  openly 
discuss  government -only  issues,  including  pending  contract 
negotiations  and  proprietary  information.  The  prime  contractors  are 
brought  in  when  the  program  manager  determines  it  is  appropriate. 

Prime  contractors  normally  are,  and  will  continue  to  be,  active 
participants  in  the  IPTs.  However,  the  recommendation  suggests  that 
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CNR  require  that  the  prime  contractors  be  present.  This  decision 
should  be  made  by  the  contracting  officer  representative/acquisition 
program  manager  at  his  discretion. 

Recommendation  A.2i 

We  recommend  that  the  Chief  of  Naval  Research,  for  advanced 
technology  development  efforts  require  the  establishment  of 
formal  memorandums  of  agreement  or  understanding  and  technology 
transition  agreements  between  the  science  and  technology  manager 
and  candidate  acquisition  program  manager ts) .  The  establishment 
of  formal  agreements  should  be  a  requirement  for  continued 
research,  development,  test,  and  evaluation  funding. 

Department  of  the  Navy  Response: 

Concur.  Ninety-two  percent  (92%)  of  the  FNCs  currently  have  formal 
MOAs  for  transitions  in  FY  2003  to  targeted  acquisition  programs.  All 
of  the  FNCs  will  have  formal  memorandums  of  agreement  (MOA)  soon.  ONR 
is  implementing  a  requirement  for  MOAs  at  the  project  level  as 
transition  arrangements  are  made.  ONR  is  devoting  much  attention  to 
determining  at  what  level  of  maturity,  size  and/or  technical  detail 
MOAs  should  be  required,  and  with  whom.  It  is  imprudent  to  impose  a 
blanket  requirement  for  an  MOA  in  every  6.3  project,  or  to  require 
management  to  stop  a  project  prematurely  for  lack  of  one;  but 
inability  to  reach  an  MOA  with  an  acquisition  program  becomes  an 
indicator  of  a  transition  problem  as  a  project  matures.  FNCs  will 
document  compliance  with  this  recommendation  in  the  ONR  Investment 
Balance  Review  and  management  controls  statements  Of  assurance  for  FY 
2003. 

Recommendation  A.3i 

We  recommend  that  the  Chief  of  Naval  Research,  for  advanced 
technology  development  efforts  review  the  technology  paths  or 
plans  for  the  5  unfunded  science  and  technology  products  that 
are  scheduled  to  transition  in  FY  2002  and  2003  and  discontinue 
any  product  development  that  does  not  have  formal  acquisition 
program  support  with  identified  funding. 

Department  of  the  Navy  Response: 


Non-concur.  Recommend  removal  of  the  word  "formal*  and  the  phrase 
"with  identified  funding." 

The  Navy  understands  that  the  recommendation  literally  applies 
only  to  unfunded  projects  scheduled  for  transition  in  FY  02-03,  hut 
the  clear  intent  of  the  report’s  findings  (see  p.10)  is  that  the 
principle  should  be  applied  in  future  cases.  The  Navy  concurs  with 
the  concerns  that  the  DDR4E  expressed  to  the  DCSD  (AT&L)  in  response 
to  a  similar  recommendation  regarding  Array  6.3  programs  (attachment 
(1) ) .  The  entire  effort  to  bring  transformational  technologies  to  the 
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operating  forces  depends  on  an  acquisition  environment  that,  as  stated 
by  USD  (AT&L) ,  "fosters  efficiency,  creativity,  and  innovation."  Such 
an  environment,  in  turn,  depends  on  the  ability  to  use  €.3  funds  to 
demonstrate  unproven  technologies,  and  explore  risk  mitigation,  before 
an  acquisition  program  commits  to  a  technology.  The  Chief  of  Naval 
Research  has  no  authority  to  compel  an  acquisition  program  manager  to 
identify  transition  funding.  Therefore,  all  6.3  programs  that  had  not 
already  satisfied  the  criteria  for  acquisition  would  have  to  be 
discontinued  in  order  to  comply  with  this  recommendation.  Such  action 
would  effectively  move  6.3  efforts  past  milestone  A  and  severely 
hamper  ONR's  efforts  to  provide  innovative  solutions  and 
transformational  naval  capabilities. 

Recommendation  B: 

We  recommend  that  the  Chief  of  Naval  Research  require  that 
the  personnel  performance  plans  for  product  managers  responsible 
for  advanced  technology  development-funded  programs  explicitly 
require  a  supervisor's  assessment  of  the  manager's  performance 
with  planned  technology  users.  The  performance  plans  should 
include  performance  assessments  for  establishment  of  working- 
level  integrated  product  teams  with  all  planned  (primary  and 
secondary)  technology  users,  creation  of  integrated  product  team 
charters ,  coordination  and  acceptance  of  quantitative  metrics  and 
key  exit  criteria  with  all  planned  users,  development  of 
transition  plans  that  are  formally  agreed  to  by  all  planned 
users,  and  development  and  maintenance  of  up-to-date  memorandums 
of  agreement  or  understanding. 

Department  of  the  Navy  Response: 

Concur.  ONR  is  revising  its  personnel  performance  system  to  a 
demonstration  model  that  is  being  tested  in  FY03  and  scheduled  for 
implementation  in  FY04.  Performance  standards  for  all  positions  in 
the  two  senior  bands  of  the  Science  and  Engineering  Career  track  (that 
is,  above  the  GS-11  level)  contain  a  Critical  Element  for  Program 
Planning  and  Leadership,  which  includes  the  following  standard: 

"Work  results  in  significant  new  scientific  and/or  technical 
results  as  well  as  applications  or  other  outcomes  that 
significantly  expand  the  future  capabilities  of  the  Navy  and 
Marine  Corps . " 

This  standard  will  be  implemented  whether  or  not  the 
demonstration  model  is  approved.  Performance  against  this  standard 
will  be  documented  in  FY  2003  Yearly  Accomplishment  Reports  and 
evaluations.  Additionally,  ONR  has  begun  a  database  of  organizational 
metrics  (discussed  below)  which  includes  transitions  as  a  major 
indicator  of  success.  Data  has  been  collected  for  FY  2001,  and  FY 
2002  data  is  due  for  collection  in  January  2003. 
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Recotranenda t i on  C.l: 


Me  recommend  that  the  Chief  of  Naval  Research  require  Science 
and  Technology  Department  officials  responsible  for  technology 
transition  programs  to  identify  and  document  appropriate 
management  controls  that  are  measurable  and  that  will  contribute 
to  preparing  the  annual  statement  of  assurance. 

Department  of  the  Navy  Response: 

Concur.  ONR  formed  an  S&T  Performance  Measurement  Analysis  Program  in 
January  1999,  established  performance  measurement  prototypes  in  August 
1999,  convened  a  Working  Group  in  July  2001,  conducted  a  Balanced 
Scorecard  study  of  vendors  in  November  2001,  and  developed  three  web- 
based  performance  measurement  prototypes  for  test  and  evaluation  in 
April  2002.  Preliminary  data  have  been  collected  for  FY  2001,  and  are 
due  for  FY  2002  in  January  2003.  One  of  the  most  significant  metric 
is  "Transitions, "  defined  as  "S&T  that  is  sufficiently  matured  to  the 
point  where  a  product  has  moved  on  and  somebody  is  doing  something 
with  it  (i.e.,  they  bought  it)."  Compliance  with  this  recommendation 
will  be  documented  in  the  next  statement  of  assurance  in  September 
2003. 

Recommendation  C.2.- 

Me  recommend  that  the  Chief  of  Naval  Research  direct  the 
Corporate  Business  Council  to  conduct  annual  command  reviews 
and  use  the  review  results  as  a  basis  for  the  Office  of  Naval 
Research  annual  statement  of  assurance. 

Department  of  the  Navy  Response; 

Concur.  ONRINST  5200. IB  assigns  responsibility  to  the  ONR 
Headquarters  (HQ)  Corporate  Business  Council  (CBC)  for  reviewing 
compliance  with  ONRINST  5200. IB  and  with  SECNAVINST  5200. 35D  for  ONR 
Corporate  Staff  elements  and  Departments.  These  reviews  should  be 
conducted  such  that  about  one-third  of  the  Corporate  Staff  elements 
and  Departments  are  reviewed  each  year  and  no  department  is  reviewed 
at  greater  than  three  year  intervals.  During  FY  2002,  the  CBC 
reviewed  59  of  ONR' s  138  assessable  units.  Hie  results  of  these 
reviews  contributed  to  ONR's  FY  2002  annual  assurance  statement. 

These  CBC  reviews  will  continue  in  the  manner  prescribed  by  ONRINST 
5200. IB  with  the  results  contributing  to  ONR's  annual  statement  of 
assurance. 

Recommendation  C ■ 3 : 

We  recommend  that  the  Chief  of  Naval  Research  provide  management 
control  training  to  the  Science  and  Technology  officials, 
consistent  with  the  requirements  of  DOD  Directive  5010. 3B. 
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FinaJ  Report 
Reference 


Page  28 
Page  29 
Page  30 


Page  32 


Department  of  the  Navy  Response: 

Concur.  ONR  will  continue  to  provide  training  to  ONR  managers 
consistent  with  their  responsibilities  and  obligations .  During  FY 
2002  ONR  developed  web-based  management  control  training  based  upon 
the  requirements  of  DoD  Directive  5010.38.4.4:  "In  the  most  cost- 
effective  manner,  the  DoD  Components  should  provide  managers 
throughout  their  organization  with  training  consistent  with  their  MC 
responsibilities  and  obligations.*;  and  SECNAVINST  5200. 35D,  Enclosure 
(2).l.f.  "Ensure  that  subordinate  commanders/managers  are  provided 
with  appropriate  training  concerning  their  Management  Control  Program 
responsibilities.*  ONR's  web-based  training  covers  pertinent 
SECNAVINST  5200. 35D  requirements.  It  is  provided  in  a  web-based 
format  in  recognition  of  the  need  to  have  a  means  of  delivery 
available  on  demand.  This  training  is  still  augmented  by  assistance 
from  the  Management  Control  Coordinator  when  needed,  required,  or 
requested . 

Appendix  A,  Material  Management  Control  Weakness i 

Adequacy  of  Management  Controls.  We  identified  a  material 
management  control  weakness  for  ONR  as  defined  by  DOD  Instruction 
5010.40.  ONR  management  controls  for  technology  transitioning 
within  the  S&T  Directorate  were  not  adequate  to  provide 
reasonable  assurance  that  the  transition  effort  had  been  executed 
efficiently  and  effectively.  Recommendations  C.l,  c.2  and  C.3, 
if  implemented,  will  improve  ONR  management  controls  over  the 
technology  transition  effort.  A  copy  of  the  report  will  be 
provided  to  the  senior  Navy  official  responsible  for  management 
controls. 

Department  of  the  Navy  Response: 

Concur.  Compliance  with  recommendations  C.l,  C.2.,  and  C.3.  will  be 
included  in  the  next  annual  statement  of  assurance  in  September  2003. 

Appendix  C: 

Department  of  the  Navy  Response: 

Page  24.  The  correct  transition  date  for  Image  Processing  and 
(  Exploitation  Architecture  is  FY  2005.  The  correct  date  for  Integrated 
Decision  Support  Suite  is  FY  2004. 

Page  25.  The  correct  transition  dates  for  Multinational  Virtual 
Operations  Capabilities  are  FY  2004  and  FY  2007. 

Page  26.  The  correct  transition  date  for  the  On-hull  Extremely  Low 
Frequency  Antenna  is  FY  2005. 

Page  28.  A  signed  MQA  for  the  Lightweight  Broadband  Variable  Depth 
Sonar  was  provided  to  the  auditors .  The  column  should  read  "yea . " 
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Sample  list  of  Recent  Projects  that  Did  Not  Have  Acquisition  Sponsorship 


Predator/Global  Hawk  -  concept  Of  unmanned  aerial  vehicles  unacceptable  at  the  time  no 
PE 

Portal  Shield  -  basis  of  many  deployed  systems  (or  chemTbio  detection  si  bases  all  over 
world  including  Pentagon.  Had  no  PE  at  the  lime. 

Automated  Deep  Operations  Coordination  System  -  Joint  C4JSR  currently  has  no  PE 
and  no  home  now  but  CENTCOM  raves  about  its  capability  to  allow  much  faster 
accurate  decision  making. 

C4t  for  Coalition  Warfare  -  Automating  ATO  transfer  between  US  and  UK  „.  no  PE. 

Personnel  Recovers  Million  Software  -  Joint  Search  and  Rescue  applications  didn't  have 
a  PE  at  the  time. 

Rapid  Terrain  Visualhation  -  Still  no  PE  to  accept  this  powerful  processing  of 
geospatial  imagery  for  fast  and  accurate  situational  awareness,  targeting  and  rebuilding  of 
the  Pentagon 

Situation  Awareness  Data  Link  -  Had  so  PH  in  provide  3-D  pictuie  of  potential  targets 
passed  between  multi  Service  aircraft 

Area  Cruise  Missile  defense  -  No  PE  for  joint  composite  tracking  "Network  Based 
Expeditionary  Command  Center"  software  that  integrates  radar  systems  and  fuses  H«« 
from  them  as  well  as  airborne,  land  and  sea-based  sensors.  Now  being  used  to  execute 
the  CAP  at  the  I"  Air  Force! 

Stealth  Technology  -  Original  stealth  Technology  was  not  dependent  upon  acquisition 
program 

Interactive  Muttisensor  Analysis  Trainiar  tIMAT)  -  IMAT  was  a  63  funded  program 
developed  for  teaching  underlying  physic*  for  ASW  in  Aviation  Warfare  Operator  (AW) 
Apprentice  School.  IMAT  was  successfully  extended  u>  Apprentice.  Advanced  and 
Officer  Courses  in  Air,  Surface,  and  Sub  Communities.  The  visualization  and  Sensor 
Performance  Modeling  technologies  became  die  lursis  for  next-generation  tactical 
displays  in  submarine  AN/BQQ-10  (ARCI)  enmbat  system,  AN/SQQ-89  Surface  Ship 
Sonar  System,  and  IUSS.  In  addition  to  being  used  for  training,  IMAT  is  now  a  fleet- 
approved  tactical  decision  aid  on  all  submarines  and  surface  ships,  DESKONS,  and 
CTRs.  It  is  IT-21  certified  for  use  on  afloat  systems  and  networks. 

Virtual  At  Sea  Training  (VAST)  -  Expense  and  uncertainties  in  scheduling  and  range 
availability  make  live-fire  training  for  naval  forces  against  realistic  land-based  target 
ranges  difficult  to  guarantee  and  costly  to  sustain.  The  solution  was  to  develop 
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technology  incorporating  live-fire  exercises  with  virtual  reality  displays  so  that  Naval 
Surface  Fire  Support  (NSFS)  training  cm  be  conducted  for  ships'  crews  and  IJSMC 
forward  observers  anywhere  at  sea.  ONR  invested  f>.3  funds  to  integrate  live  and  virtual 
systems  that  became  die  backbone  of  the  VAST  system.  Atlantic  and  Pacific  Fleets  have 
requested  a  minimum  of  17  Advanced  NSFS  VAST  systems  in  the  FYOd  timeframe. 


Streak  Tube  I  marine  UP  AH  -  This  6J  program  was  sinned  in  l'Y97  lo  demonstrate 
underwater  imaging  using  an  ONR  developed  (6.2)  streak  tube  imaging  lechnologj.  At 
the  time,  there  was  no  acquisition  program,  or  programmatic  interest  in  rapid  mine 
identification.  As  a  result  of  this  work  (along  with  ONR  funded  research  in  laser  tine 
scan  technology)  an  acquisition  program  was  established  in  FY02.  This  technology  has 
transitioned  to  two  acquisition  programs  f  AOS  Ot)  and  A  LMDS  ). 

Laser  Lint  Scan  imatinr  technology  dtmonstratio) t  -  This  6  J  program  was 
successfully  demoostnued  during  KBW  and  Hcri  Barite  Rxpcriment  Hotel.  Navy  Ins 
now  procured  5  systems  that  me  currently  in  service  (AQS-ldB). 

A  I/Mr  for  very  shallow  water  —  This  6.3  program  development  and  demonstration  began 
in  FY99.  As  a  resuh  of  this  program,  a  shallow  water  AUV  acquisition  program  was 
initiated  during  PRO  I  budget  process. 

Airborne  Laser  —  The  original  work  came  out  of  the  Air  Race  Airborne  Laser 
Laboratory — a  6.2  and  6.3  activity  to  demonstrate  the  feasibility  of  laser  on  a  platform 
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